UPI/Bare Metal - The master node cannot finish the installation ("config/master", "timeout awaiting response headers"/"EOF") #696

eduardolucioac · 2021-06-17T02:34:12Z

eduardolucioac
Jun 17, 2021

Simple question, but so far very difficult to answer... =-[

I am trying to deploy OpenShift (OKD) 4.5 or 4.7 as directed here Guide: Installing an OKD 4.5 Cluster. Look at the "Starting the control plane nodes" section.

I'm trying to create the cluster using an UPI (User Provisioned Infrastructure)/Bare Metal (KVM).

PROBLEM:

Version 4.5

The "master" node cannot finish installation after reboot. It keeps showing the following error...

[ 1304.254380] ignition[485]: GET https://api-int.mbr.okd.local:22623/config/master: attempt #92
[ 1314.264629] ignition[485]: GET error: Get "https://api-int.mbr.okd.local:22623/config/master": net/http: timeout awaiting response headers

For version 4.5 we use "Fedora CoreOS 32.20200715.3.0".

Version 4.7

The "master" node cannot finish installation after reboot. It keeps showing the following error...

[  543.933709] ignition[505]: GET https://api-int.mbr.okd.local:22623/config/master: attempt #112
[  543.939340] ignition[505]: GET error: Get "https://api-int.mbr.okd.loca1:22623/config/master": EOF

For version 4.7 we use "Fedora CoreOS 34.20210518.3.0".

I've waited hours and hours and the master nodes are still in the same situation. What can I do to resolve this?

Thanks! =D

eduardolucioac · 2021-06-17T13:44:11Z

eduardolucioac
Jun 17, 2021
Author

The state of the bootstrap node for more than 10 hours...

[root@okd_services ~]# openshift-install wait-for bootstrap-complete --log-level=info --dir=/usr/local/okd/
INFO Waiting up to 20m0s for the Kubernetes API at https://api.mbr.okd.local:6443... 
INFO API v1.18.3 up                               
INFO Waiting up to 40m0s for bootstrapping to complete...

New error on master node...

[46016.424428] ignition[485]: GET https://api-int.mbr.okd.local:22623/config/master: attempt #8241
[46016.429405] ignition[485]: GET error: Get "https://api-int.mbr.okd.local:22623/config/master": dial tcp: lookup api-int.mbr.okd.local on 10.3.0.14:53: dial udp 10.3.0.14:53: connect: network is unreachable

Any, help!? =D

4 replies

vrutkovs Jun 17, 2021
Maintainer

dial udp 10.3.0.14:53: connect: network is unreachable

Invalid network configuration on masters, see https://docs.okd.io/latest/installing/installing_bare_metal/installing-bare-metal.html#installation-network-user-infra_installing-bare-metal

eduardolucioac Jun 17, 2021
Author

All DHCP and DNS configurations are correct, as all machines have IP, "see" each other and access the WAN.

Information or tools to help diagnose what is going on are most welcome! Do you have any suggestions or ideas?

Thanks! =D

vrutkovs Jun 17, 2021
Maintainer

"see" each other and access the WAN

Clearly, the machines don't see bootstrap node or the LB serving api-int

eduardolucioac Jun 17, 2021
Author

See if this helps...

This output occurs in okd_master_3 (10.3.0.7)....

[ 1304.254380] ignition[485]: GET https://api-int.mbr.okd.local:22623/config/master: attempt #92
[ 1314.264629] ignition[485]: GET error: Get "https://api-int.mbr.okd.local:22623/config/master": net/http: timeout awaiting response headers

Connecting okd_master_2 (10.3.0.6) from okd_services (10.3.0.14)...

NOTE: The okd_master_2 (10.3.0.6) was able to boot (reached login screen).

[root@okd_services ~]# ssh [email protected]
The authenticity of host '10.3.0.6 (10.3.0.6)' can't be established.
ECDSA key fingerprint is SHA256:1xdq65g0ljnZYR6uXHaXW6EsxO3u6X268s4Z9Kfq0ng.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.3.0.6' (ECDSA) to the list of known hosts.
Fedora CoreOS 32.20200629.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

Pinging the okd_bootstrap (10.3.0.4) from okd_master_2 (10.3.0.6)...

[core@localhost ~]$ ping 10.3.0.4
PING 10.3.0.4 (10.3.0.4) 56(84) bytes of data.
64 bytes from 10.3.0.4: icmp_seq=1 ttl=64 time=0.973 ms
64 bytes from 10.3.0.4: icmp_seq=2 ttl=64 time=0.801 ms
64 bytes from 10.3.0.4: icmp_seq=3 ttl=64 time=0.373 ms
64 bytes from 10.3.0.4: icmp_seq=4 ttl=64 time=0.647 ms
^C
--- 10.3.0.4 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3032ms
rtt min/avg/max/mdev = 0.373/0.698/0.973/0.220 ms

Calling the problematic URL from okd_master_2 (10.3.0.6)...

[core@localhost ~]$ curl -kv https://api-int.mbr.okd.local:22623/config/master
*   Trying 10.3.0.14:22623...
* Connected to api-int.mbr.okd.local (10.3.0.14) port 22623 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api-int.mbr.okd.local
*  start date: Jun 16 23:52:22 2021 GMT
*  expire date: Jun 14 23:52:23 2031 GMT
*  issuer: OU=openshift; CN=root-ca
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561ed249aa40)
> GET /config/master HTTP/2
> Host: api-int.mbr.okd.local:22623
> user-agent: curl/7.69.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 500 
< content-length: 0
< date: Thu, 17 Jun 2021 14:55:43 GMT
< 
* Connection #0 to host api-int.mbr.okd.local left intact

ABOUT THE INFRASTRUCTURE:

Virtual machines...

NAME           ROLE                   OS              IP          MAC
okd_boostrap   bootstrap              Fedora CoreOS   10.3.0.4    52:54:00:07:80:62
okd_master_1   master                 Fedora CoreOS   10.3.0.5    52:54:00:7d:97:70
okd_master_2   master                 Fedora CoreOS   10.3.0.6    52:54:00:6e:52:85
okd_master_3   master                 Fedora CoreOS   10.3.0.7    52:54:00:a3:65:d9
okd_worker_1   worker                 Fedora CoreOS   10.3.0.8    52:54:00:e3:7c:fb
okd_worker_2   worker                 Fedora CoreOS   10.3.0.9    52:54:00:20:ec:4f
okd_services   DNS/LB/web/NFS         CentOS 8        10.3.0.14   52:54:00:3a:fd:a2
                                                         10.2.0.18   52:54:00:92:ce:78
okd_pfsense    firewall/router/DHCP   FreeBSD         10.3.0.2 52:54:00:d8:27:82
                                                         10.2.0.19   52:54:00:ac:82:7d

 . OKD_LAN: "10.3.0";
 . EXT_LAN: "10.2.0".

Some acronyms...
 _ DNS - Domain Name System;
 _ LB - Load Balancing;
 _ Web - Web Server;
 _ NFS - Network File Sharing.

Network layout...

           ...→.[N]WAN/EXT_LAN([R]dhcp).←... (10.2.0.0/24)
           ↓                               ↓
          [I]WAN/EXT_LAN                  [I]WAN/EXT_LAN
  [V]OKD_PFSENSE([R]dhcp)                 [V]OKD_SERVICES
          [I]OKD_LAN                      [I]OKD_LAN
           ↑                               ↑
           .........→.[N]OKD_LAN.←.......... (10.3.0.0/24)
                       ↑
      ...................................
      ↓                ↓                ↓
     [V]OKD_BOOSTRAP  [V]OKD_MASTER_1  [V]OKD_WORKER_1
                      [V]OKD_MASTER_2  [V]OKD_WORKER_2
                      [V]OKD_MASTER_3

 _ [N] - Network;
 _ [R] - Network Resource;
 _ [I] - Network Interface;
 _ [V] - Virtual Machine.

Thanks! =D

eduardolucioac · 2021-06-17T15:28:21Z

eduardolucioac
Jun 17, 2021
Author

See if this helps...

This output occurs in okd_master_3 (10.3.0.7)....

[ 1304.254380] ignition[485]: GET https://api-int.mbr.okd.local:22623/config/master: attempt #92
[ 1314.264629] ignition[485]: GET error: Get "https://api-int.mbr.okd.local:22623/config/master": net/http: timeout awaiting response headers

Connecting okd_master_2 (10.3.0.6) from okd_services (10.3.0.14)...

NOTE: The okd_master_2 (10.3.0.6) was able to boot (reached login screen).

[root@okd_services ~]# ssh [email protected]
The authenticity of host '10.3.0.6 (10.3.0.6)' can't be established.
ECDSA key fingerprint is SHA256:1xdq65g0ljnZYR6uXHaXW6EsxO3u6X268s4Z9Kfq0ng.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.3.0.6' (ECDSA) to the list of known hosts.
Fedora CoreOS 32.20200629.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

Pinging the okd_bootstrap (10.3.0.4) from okd_master_2 (10.3.0.6)...

[core@localhost ~]$ ping 10.3.0.4
PING 10.3.0.4 (10.3.0.4) 56(84) bytes of data.
64 bytes from 10.3.0.4: icmp_seq=1 ttl=64 time=0.973 ms
64 bytes from 10.3.0.4: icmp_seq=2 ttl=64 time=0.801 ms
64 bytes from 10.3.0.4: icmp_seq=3 ttl=64 time=0.373 ms
64 bytes from 10.3.0.4: icmp_seq=4 ttl=64 time=0.647 ms
^C
--- 10.3.0.4 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3032ms
rtt min/avg/max/mdev = 0.373/0.698/0.973/0.220 ms

Calling the problematic URL from okd_master_2 (10.3.0.6)...

[core@localhost ~]$ curl -kv https://api-int.mbr.okd.local:22623/config/master
*   Trying 10.3.0.14:22623...
* Connected to api-int.mbr.okd.local (10.3.0.14) port 22623 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api-int.mbr.okd.local
*  start date: Jun 16 23:52:22 2021 GMT
*  expire date: Jun 14 23:52:23 2031 GMT
*  issuer: OU=openshift; CN=root-ca
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561ed249aa40)
> GET /config/master HTTP/2
> Host: api-int.mbr.okd.local:22623
> user-agent: curl/7.69.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 500 
< content-length: 0
< date: Thu, 17 Jun 2021 14:55:43 GMT
< 
* Connection #0 to host api-int.mbr.okd.local left intact

ABOUT THE INFRASTRUCTURE:

Virtual machines...

NAME           ROLE                   OS              IP          MAC
okd_boostrap   bootstrap              Fedora CoreOS   10.3.0.4    52:54:00:07:80:62
okd_master_1   master                 Fedora CoreOS   10.3.0.5    52:54:00:7d:97:70
okd_master_2   master                 Fedora CoreOS   10.3.0.6    52:54:00:6e:52:85
okd_master_3   master                 Fedora CoreOS   10.3.0.7    52:54:00:a3:65:d9
okd_worker_1   worker                 Fedora CoreOS   10.3.0.8    52:54:00:e3:7c:fb
okd_worker_2   worker                 Fedora CoreOS   10.3.0.9    52:54:00:20:ec:4f
okd_services   DNS/LB/web/NFS         CentOS 8        10.3.0.14   52:54:00:3a:fd:a2
                                                         10.2.0.18   52:54:00:92:ce:78
okd_pfsense    firewall/router/DHCP   FreeBSD         10.3.0.2 52:54:00:d8:27:82
                                                         10.2.0.19   52:54:00:ac:82:7d

 . OKD_LAN: "10.3.0";
 . EXT_LAN: "10.2.0".

Some acronyms...
 _ DNS - Domain Name System;
 _ LB - Load Balancing;
 _ Web - Web Server;
 _ NFS - Network File Sharing.

Network layout...

           ...→.[N]WAN/EXT_LAN([R]dhcp).←... (10.2.0.0/24)
           ↓                               ↓
          [I]WAN/EXT_LAN                  [I]WAN/EXT_LAN
  [V]OKD_PFSENSE([R]dhcp)                 [V]OKD_SERVICES
          [I]OKD_LAN                      [I]OKD_LAN
           ↑                               ↑
           .........→.[N]OKD_LAN.←.......... (10.3.0.0/24)
                       ↑
      ...................................
      ↓                ↓                ↓
     [V]OKD_BOOSTRAP  [V]OKD_MASTER_1  [V]OKD_WORKER_1
                      [V]OKD_MASTER_2  [V]OKD_WORKER_2
                      [V]OKD_MASTER_3

 _ [N] - Network;
 _ [R] - Network Resource;
 _ [I] - Network Interface;
 _ [V] - Virtual Machine.

Thanks! =D

0 replies

eduardolucioac · 2021-06-17T16:19:41Z

eduardolucioac
Jun 17, 2021
Author

Configuration files...

BIND 9 (DNS):

. db.10.3.0

$TTL    604800
@   IN  SOA okd-services.okd.local. admin.okd.local. (
        6       ; Serial
        604800  ; Refresh
        86400   ; Retry
        2419200 ; Expire
        604800  ; Negative Cache TTL
)

; Name servers - "NS" records.
    IN  NS  okd-services.okd.local.

; Name servers - "PTR" records.
14 IN  PTR okd-services.okd.local.

; OpenShift container platform cluster - "PTR" records.
4 IN  PTR okd-boostrap.mbr.okd.local.
5 IN  PTR okd-master-1.mbr.okd.local.
6 IN  PTR okd-master-2.mbr.okd.local.
7 IN  PTR okd-master-3.mbr.okd.local.
8 IN  PTR okd-worker-1.mbr.okd.local.
9 IN  PTR okd-worker-2.mbr.okd.local.
14 IN  PTR api.mbr.okd.local.
14 IN  PTR api-int.mbr.okd.local.

. db.okd.local

$TTL    604800
@   IN  SOA okd-services.okd.local. admin.okd.local. (
        1       ; Serial
        604800  ; Refresh
        86400   ; Retry
        2419200 ; Expire
        604800  ; Negative Cache TTL
)

; Name servers - "NS" records.
    IN  NS  okd-services

; Name servers - "A" records.
okd-services.okd.local. IN A 10.3.0.14

; OpenShift container platform cluster - "A" records.
okd-boostrap.mbr.okd.local. IN  A   10.3.0.4
okd-master-1.mbr.okd.local. IN  A   10.3.0.5
okd-master-2.mbr.okd.local. IN  A   10.3.0.6
okd-master-3.mbr.okd.local. IN  A   10.3.0.7
okd-worker-1.mbr.okd.local. IN  A   10.3.0.8
okd-worker-2.mbr.okd.local. IN  A   10.3.0.9

; Openshift internal cluster IPs - "A" records.
api.mbr.okd.local.              IN  A   10.3.0.14
api-int.mbr.okd.local.          IN  A   10.3.0.14
*.apps.mbr.okd.local.           IN  A   10.3.0.14
etcd-0.mbr.okd.local.           IN  A   10.3.0.5
etcd-1.mbr.okd.local.           IN  A   10.3.0.6
etcd-2.mbr.okd.local.           IN  A   10.3.0.7
cons-okd.apps.mbr.okd.local.    IN  A   10.3.0.14
oauth-okd.apps.mbr.okd.local.   IN  A   10.3.0.14

; OpenShift internal cluster IPs - "SRV" records.
_etcd-server-ssl._tcp.mbr.okd.local.    86400   IN  SRV 0   10  2380    etcd-0.mbr
_etcd-server-ssl._tcp.mbr.okd.local.    86400   IN  SRV 0   10  2380    etcd-1.mbr
_etcd-server-ssl._tcp.mbr.okd.local.    86400   IN  SRV 0   10  2380    etcd-2.mbr

. named.conf.local

zone "okd.local" {
    type master;
    file "/etc/named/zones/db.okd.local"; // Zone file path.
};

zone "0.3.10.in-addr.arpa" {
    type master;
    file "/etc/named/zones/db.10.3.0"; // 10.3.0.0/24 subnet.
};

. named.conf

//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS server
// as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
// See the BIND Administrator's Reference Manual (ARM) for details about the configuration
// located in /usr/share/doc/bind-{version}/Bv9ARM.html .

options {
    listen-on port 53 { 127.0.0.1; 10.3.0.14; };
    directory "/var/named";
    dump-file "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    recursing-file "/var/named/data/named.recursing";
    secroots-file "/var/named/data/named.secroots";
    allow-query { localhost; 10.3.0.0/24; };

    // - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
    // - If you are building a RECURSIVE (caching) DNS server, you need to enable
    // recursion.
    // - If your recursive DNS server has a public IP address, you MUST enable access
    // control to limit queries to your legitimate users. Failing to do so will cause
    // your server to become part of large scale DNS amplification attacks. Implementing
    // BCP38 within your network would greatly reduce such attack surface.
    recursion yes;

    forwarders {
        8.8.8.8;
        8.8.4.4;
    };

    dnssec-enable yes;
    dnssec-validation yes;

    // Path to ISC DLV key.
    bindkeys-file "/etc/named.root.key";

    managed-keys-directory "/var/named/dynamic";

    pid-file "/run/named/named.pid";
    session-keyfile "/run/named/session.key";
};

logging {
    channel default_debug {
        file "data/named.run";
        severity dynamic;
    };
};

zone "." IN {
    type hint;
    file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
include "/etc/named/named.conf.local";

HAProxy (load balancer):

. haproxy.cfg

#---------------------------------------------------------------------
# Global settings.
#---------------------------------------------------------------------
global
    maxconn 20000
    log /dev/log local0 info
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    user haproxy
    group haproxy
    daemon

    # Turn on stats unix socket.
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# Common defaults that all the "listen" and "backend" sections will use if not designated
# in their block.
#---------------------------------------------------------------------
defaults
    mode http
    log global
    option httplog
    option dontlognull
    option http-server-close
    option redispatch
    retries 3
    timeout http-request 10s
    timeout queue 1m
    timeout connect 10s
    timeout client 300s
    timeout server 300s
    timeout http-keep-alive 10s
    timeout check 10s
    maxconn 20000

listen stats
    bind :9000
    mode http
    option forwardfor except 127.0.0.0/8
    stats enable
    stats uri /

frontend okd_k8s_api_fe
    bind :6443
    default_backend okd_k8s_api_be
    mode tcp
    option tcplog

backend okd_k8s_api_be
    balance source
    mode tcp
    server okd-boostrap 10.3.0.4:6443 check
    server okd-master-1 10.3.0.5:6443 check
    server okd-master-2 10.3.0.6:6443 check
    server okd-master-3 10.3.0.7:6443 check

frontend okd_machine_config_server_fe
    bind :22623
    default_backend okd_machine_config_server_be
    mode tcp
    option tcplog

backend okd_machine_config_server_be
    balance source
    mode tcp
    server okd-boostrap 10.3.0.4:22623 check
    server okd-master-1 10.3.0.5:22623 check
    server okd-master-2 10.3.0.6:22623 check
    server okd-master-3 10.3.0.7:22623 check

frontend okd_http_ingress_traffic_fe
    bind :80
    default_backend okd_http_ingress_traffic_be
    mode tcp
    option tcplog

backend okd_http_ingress_traffic_be
    balance source
    mode tcp
    server okd-worker-1 10.3.0.8:80 check
    server okd-worker-2 10.3.0.9:80 check

frontend okd_https_ingress_traffic_fe
    bind *:443
    default_backend okd_https_ingress_traffic_be
    mode tcp
    option tcplog

backend okd_https_ingress_traffic_be
    balance source
    mode tcp
    server okd-worker-1 10.3.0.8:443 check
    server okd-worker-2 10.3.0.9:443 check

OpenShift (OKD) "*.yaml" files:

. htpasswd_provider.yaml

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - name: htpasswd_provider
    mappingMethod: claim
    type: HTPasswd
    htpasswd:
      fileData:
        name: htpass-secret

. install-config.yaml

apiVersion: v1
baseDomain: okd.local
metadata:
  name: mbr

compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0

controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3

networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16

platform:
  none: {}

fips: false

pullSecret: '{"auths":{"fake":{"auth": "bar"}}}'
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAA<SKIPPED>QbAKPwwhdCkTpd8= root@okd_services.my_domain.com.br'

. registry_pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry-pv
spec:
  capacity:
    storage: 45Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /var/nfsshare/registry
    server: 10.3.0.14

Thanks! =D

1 reply

bobby0724 Jun 17, 2021

Can you run a
netstat -natup on your service machine the one that have the haproxy running and paste it here

and can you try changing the LB to

balance roundrobin

eduardolucioac · 2021-06-17T16:47:08Z

eduardolucioac
Jun 17, 2021
Author

@bobby0724

Done! =D

[root@okd_services ~]# netstat -natup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      906/sshd            
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      929/named           
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      4572/haproxy        
tcp        0      0 0.0.0.0:22623           0.0.0.0:*               LISTEN      4572/haproxy        
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      4572/haproxy        
tcp        0      0 0.0.0.0:6443            0.0.0.0:*               LISTEN      4572/haproxy        
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4572/haproxy        
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1742/dnsmasq        
tcp        0      0 10.3.0.14:53            0.0.0.0:*               LISTEN      929/named           
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      929/named           
tcp        0      0 10.2.0.18:22            10.2.0.3:44536          ESTABLISHED 1854/sshd: root [pr 
tcp        0      0 10.3.0.14:52252         10.3.0.4:6443           ESTABLISHED 4572/haproxy        
tcp        0      0 10.3.0.14:52134         10.3.0.4:6443           ESTABLISHED 4572/haproxy        
tcp        0      1 10.3.0.14:42222         10.3.0.8:443            SYN_SENT    4572/haproxy        
tcp        0      0 10.3.0.14:6443          10.3.0.6:51962          ESTABLISHED 4572/haproxy        
tcp        0      0 10.3.0.14:52130         10.3.0.4:6443           ESTABLISHED 4572/haproxy        
tcp        0      0 10.3.0.14:6443          10.3.0.6:51946          ESTABLISHED 4572/haproxy        
tcp        0      1 10.3.0.14:40530         10.3.0.9:443            SYN_SENT    4572/haproxy        
tcp        0    196 10.2.0.18:22            10.2.0.3:44538          ESTABLISHED 5000/sshd: root [pr 
tcp        0      0 10.2.0.18:45472         10.2.0.5:389            ESTABLISHED 878/sssd_be         
tcp        0      0 10.3.0.14:51970         10.3.0.4:6443           ESTABLISHED 4572/haproxy        
tcp        0      0 10.3.0.14:54056         10.3.0.4:6443           ESTABLISHED 4572/haproxy        
tcp        0      0 10.2.0.18:33328         147.75.69.225:80        TIME_WAIT   -                   
tcp        0      0 10.3.0.14:6443          10.3.0.5:39976          ESTABLISHED 4572/haproxy        
tcp        0      0 10.3.0.14:6443          10.3.0.5:52462          ESTABLISHED 4572/haproxy        
tcp        0      1 10.3.0.14:41396         10.3.0.7:22623          SYN_SENT    4572/haproxy        
tcp        0      1 10.3.0.14:41964         10.3.0.9:80             SYN_SENT    4572/haproxy        
tcp        0      1 10.3.0.14:60674         10.3.0.7:6443           SYN_SENT    4572/haproxy        
tcp        0      0 10.3.0.14:6443          10.3.0.5:40024          ESTABLISHED 4572/haproxy        
tcp        0      0 10.2.0.18:43394         109.205.222.4:80        TIME_WAIT   -                   
tcp6       0      0 :::22                   :::*                    LISTEN      906/sshd            
tcp6       0      0 ::1:953                 :::*                    LISTEN      929/named           
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::8080                 :::*                    LISTEN      1131/httpd          
tcp6       0      0 :::53                   :::*                    LISTEN      929/named           
udp        0      0 192.168.122.1:53        0.0.0.0:*                           1742/dnsmasq        
udp        0      0 10.3.0.14:53            0.0.0.0:*                           929/named           
udp        0      0 127.0.0.1:53            0.0.0.0:*                           929/named           
udp        0      0 0.0.0.0:67              0.0.0.0:*                           1742/dnsmasq        
udp        0      0 10.3.0.14:68            10.3.0.2:67             ESTABLISHED 893/NetworkManager  
udp        0      0 10.2.0.18:68            10.2.0.2:67             ESTABLISHED 893/NetworkManager  
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/systemd           
udp        0      0 127.0.0.1:323           0.0.0.0:*                           857/chronyd         
udp6       0      0 :::53                   :::*                                929/named           
udp6       0      0 :::111                  :::*                                1/systemd           
udp6       0      0 ::1:323                 :::*                                857/chronyd

1 reply

bobby0724 Jun 17, 2021

Ok the HAproxy is listening, did you change config to roundrobin ?
can you ssh the bootstrap and ping the master ?

eduardolucioac · 2021-06-17T19:55:36Z

eduardolucioac
Jun 17, 2021
Author

@bobby0724

Yes! I can ping the entire infrastructure.

Pings performed from okd_boostrap (10.3.0.4)...

[core@localhost ~]$ ping -c 4 10.3.0.5 # okd_master_1
PING 10.3.0.5 (10.3.0.5) 56(84) bytes of data.
64 bytes from 10.3.0.5: icmp_seq=1 ttl=64 time=0.292 ms
64 bytes from 10.3.0.5: icmp_seq=2 ttl=64 time=0.442 ms
64 bytes from 10.3.0.5: icmp_seq=3 ttl=64 time=0.320 ms
64 bytes from 10.3.0.5: icmp_seq=4 ttl=64 time=0.298 ms

--- 10.3.0.5 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3063ms
rtt min/avg/max/mdev = 0.292/0.338/0.442/0.060 ms
[core@localhost ~]$ ping -c 4 10.3.0.6 # okd_master_2
PING 10.3.0.6 (10.3.0.6) 56(84) bytes of data.
64 bytes from 10.3.0.6: icmp_seq=1 ttl=64 time=3.11 ms
64 bytes from 10.3.0.6: icmp_seq=2 ttl=64 time=0.347 ms
64 bytes from 10.3.0.6: icmp_seq=3 ttl=64 time=0.380 ms
64 bytes from 10.3.0.6: icmp_seq=4 ttl=64 time=0.214 ms

--- 10.3.0.6 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3063ms
rtt min/avg/max/mdev = 0.214/1.012/3.107/1.211 ms
[core@localhost ~]$ ping -c 4 10.3.0.7 # okd_master_3
PING 10.3.0.7 (10.3.0.7) 56(84) bytes of data.
64 bytes from 10.3.0.7: icmp_seq=1 ttl=64 time=0.324 ms
64 bytes from 10.3.0.7: icmp_seq=2 ttl=64 time=0.684 ms
64 bytes from 10.3.0.7: icmp_seq=3 ttl=64 time=0.364 ms
64 bytes from 10.3.0.7: icmp_seq=4 ttl=64 time=0.936 ms

--- 10.3.0.7 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
rtt min/avg/max/mdev = 0.324/0.577/0.936/0.249 ms
[core@localhost ~]$ ping -c 4 10.3.0.8 # okd_worker_1
PING 10.3.0.8 (10.3.0.8) 56(84) bytes of data.
64 bytes from 10.3.0.8: icmp_seq=1 ttl=64 time=0.379 ms
64 bytes from 10.3.0.8: icmp_seq=2 ttl=64 time=0.264 ms
64 bytes from 10.3.0.8: icmp_seq=3 ttl=64 time=0.357 ms
64 bytes from 10.3.0.8: icmp_seq=4 ttl=64 time=0.421 ms

--- 10.3.0.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
rtt min/avg/max/mdev = 0.264/0.355/0.421/0.057 ms
[core@localhost ~]$ ping -c 4 10.3.0.9 # okd_worker_2
PING 10.3.0.9 (10.3.0.9) 56(84) bytes of data.
64 bytes from 10.3.0.9: icmp_seq=1 ttl=64 time=0.566 ms
64 bytes from 10.3.0.9: icmp_seq=2 ttl=64 time=0.368 ms
64 bytes from 10.3.0.9: icmp_seq=3 ttl=64 time=0.331 ms
64 bytes from 10.3.0.9: icmp_seq=4 ttl=64 time=0.360 ms

--- 10.3.0.9 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3061ms
rtt min/avg/max/mdev = 0.331/0.406/0.566/0.093 ms
[core@localhost ~]$ ping -c 4 10.3.0.14 # okd_services
PING 10.3.0.14 (10.3.0.14) 56(84) bytes of data.
64 bytes from 10.3.0.14: icmp_seq=1 ttl=64 time=0.443 ms
64 bytes from 10.3.0.14: icmp_seq=2 ttl=64 time=0.681 ms
64 bytes from 10.3.0.14: icmp_seq=3 ttl=64 time=0.374 ms
64 bytes from 10.3.0.14: icmp_seq=4 ttl=64 time=0.509 ms

--- 10.3.0.14 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3055ms
rtt min/avg/max/mdev = 0.374/0.501/0.681/0.113 ms

0 replies

eduardolucioac · 2021-06-17T19:56:11Z

eduardolucioac
Jun 17, 2021
Author

@bobby0724

"did you change config to roundrobin" -> Yes.

=D

0 replies

eduardolucioac · 2021-06-17T20:09:37Z

eduardolucioac
Jun 17, 2021
Author

Further question...

Are these names important (below)? In other words, do they need to follow some kind of convention that is expected by OKD (OpenShift) within the infrastructure?

. BIND 9 Configurations:

api
api-int
console-openshift-console.apps
etcd-0
etcd-1
etcd-2
oauth-openshift.apps
okd4-bootstrap
okd4-compute-1
okd4-compute-2
okd4-control-plane-1
okd4-control-plane-2
okd4-control-plane-3
okd4-services
_etcd-server-ssl._tcp

. HAProxy Configurations:

okd4_https_ingress_traffic_be
okd4_https_ingress_traffic_fe
okd4_http_ingress_traffic_be
okd4_http_ingress_traffic_fe
okd4_k8s_api_be
okd4_k8s_api_fe
okd4_machine_config_server_be
okd4_machine_config_server_fe

1 reply

bobby0724 Jun 17, 2021

seems that you are following a guide right

can you share that with us so we can check

the names a important and also the reservations in the dhcp

eduardolucioac · 2021-06-17T20:46:41Z

eduardolucioac
Jun 17, 2021
Author

@bobby0724

The guide I'm following is this https://itnext.io/guide-installing-an-okd-4-5-cluster-508a2631cbee

"the names a(re) important" -> So I can't change something like "okd4-control-plane-1" to "okd-master-1" and something like "okd4_https_ingress_traffic_be" to "okd_https_ingress_traffic_be", right? (This question may sound redundant, but I really need to confirm this....)

5 replies

fengzhigu2000 Jan 12, 2022

@eduardolucioac Did you finally solve this problem? I would like to know how it is solved

eduardolucioac Jan 12, 2022
Author

@eduardolucioac Did you finally solve this problem? I would like to know how it is solved

@fengzhigu2000 The process for deploying OpenShift (OKD) is long, complex and full of details. So I strongly recommend that you follow along for the details in this guide. Most likely it will help you with more than 90% of the doubts that may arise.

Success! =D

gonhidalgo Jun 14, 2022

hey @eduardolucioac! I am also using the guide you have referenced and I am having the same issue. How did you solve it? I have also changed a bit the names of the VMs (instead of okd4-control-plane-1 i chose okd-control-1). Was that the problem?

Furthermore, I have seen that you have done your own guide, but I found a slight difference. You have not created an extra VM for the DHCP server and you have added it on the okd-services. Why is that?

I could really use your help!
Thanks!

eduardolucioac Jun 15, 2022
Author

The above error is most likely due to DNS and/or DHCP configuration issues. Note that this situation must not extend for more than 10 minutes. In any case, confirm that all machines have the minimum recommended hardware configurations (see section "Minimum hardware requirements...").

"You have not created an extra VM for the DHCP server and you have added it on the okd-services." -> Resource optimization and simplicity. There is no need for an extra machine just to provide this. Also notice that according to the scheme in "Network layout..." the OKD network (OpenShift) is totally isolated.

A hug! Success! 🤗😁🇧🇷

p0th3ad Dec 19, 2022

Hi pals,
I'm following this guide by keeping the very same subnets, names, and HW resources, but slightly different software versions:

[me@okd4-services ~]$ oc version
Client Version: 4.11.0-0.okd-2022-12-02-145640
Kustomize Version: v4.5.4
[me@okd4-services ~]$ openshift-install version
openshift-install 4.11.0-0.okd-2022-12-02-145640
built from commit c258fd3a0027ef3fa4ed685e093e044b1d4014cd
release image quay.io/openshift/okd@sha256:924958ed5ba93ea9d72e4b5a319d0d4df220edbb891816c106215cbf318a0a40
release architecture amd64

Fedora CoreOS: fedora-coreos-36.20220716.3.1-metal.x86_64.raw.xz

I also have the same problem with okd4-control-plane-1

and I'm stuck here for hours now. Rebooting services didn't help.

DHCP and DNS looks good to me. IP addresses and names are correct.
Did you fix your issue gonhidalgo?

eduardolucioac · 2021-06-17T23:53:30Z

eduardolucioac
Jun 17, 2021
Author

Further question...

Are these names important (below)? In other words, do they need to follow some kind of convention that is expected by OKD (OpenShift) within the infrastructure?

. BIND 9 Configurations:
api
api-int
console-openshift-console.apps
etcd-0
etcd-1
etcd-2
oauth-openshift.apps
okd4-bootstrap
okd4-compute-1
okd4-compute-2
okd4-control-plane-1
okd4-control-plane-2
okd4-control-plane-3
okd4-services
_etcd-server-ssl._tcp
. HAProxy Configurations:
okd4_https_ingress_traffic_be
okd4_https_ingress_traffic_fe
okd4_http_ingress_traffic_be
okd4_http_ingress_traffic_fe
okd4_k8s_api_be
okd4_k8s_api_fe
okd4_machine_config_server_be
okd4_machine_config_server_fe

I adjusted the names according to the "defaults" above and now all machines (nodes) reach the login splash screen.

One of the nodes I had to redo the process (reinstall) to work.

However, now we have two problems...

1# The nodes are all the time printing information on the screen that is mixed with the traditional components of the initial login screen. I don't know if this is normal...;

2# According to the command below the bootstrap process is not finishing...

[root@okd_services ~]# openshift-install wait-for bootstrap-complete --log-level=info --dir=/usr/local/okd/
INFO Waiting up to 20m0s for the Kubernetes API at https://api.mbr.okd.local:6443... 
INFO API v1.18.3 up                               
INFO Waiting up to 40m0s for bootstrapping to complete...

... so it doesn't go over the line "INFO Waiting up to 40m0s for bootstrapping to complete...".

Further question...

The fact that I use this configuration...

pullSecret: '{"auths":{"fake":{"auth": "bar"}}}'

... could it be cause of a problem?

0 replies

eduardolucioac · 2021-06-18T16:02:05Z

eduardolucioac
Jun 18, 2021
Author

"1# The nodes are all the time printing information on the screen that is mixed with the traditional components of the initial login screen. I don't know if this is normal...;" -> An image may help...

0 replies

bobby0724 · 2021-06-18T16:14:19Z

bobby0724
Jun 18, 2021

Yes this means the server have finally connect to the bootstrap and the configuration have started
the only problem is that the bootstrap is only valid for 24 hours
you will need to start over fresh install

0 replies

fengzhigu2000 · 2022-01-12T15:06:11Z

fengzhigu2000
Jan 12, 2022

I also encountered the same problem, but I used the form of static configuration IP, I added kernel parameters through Fedora-coreos->tag

0 replies

eduardolucioac · 2022-12-19T20:11:06Z

eduardolucioac
Dec 19, 2022
Author

Don't waste time with the guide you provided. If you want to install OpenShift (OKD) 4.X cluster over UPI/"bare-metal". This guide is the way to go! Install the OpenShift (OKD) 4.X cluster (UPI/"bare-metal") <https://github.com/eduardolucioac/okd_bare_metal> 😉 Em seg., 19 de dez. de 2022 às 16:02, p0th3ad ***@***.***> escreveu:

…

Hi pals, I'm following this guide <https://itnext.io/guide-installing-an-okd-4-5-cluster-508a2631cbee> by keeping the very same subnets and names, but slightly different versions: ***@***.*** ~]$ oc version Client Version: 4.11.0-0.okd-2022-12-02-145640 Kustomize Version: v4.5.4 ***@***.*** ~]$ openshift-install version openshift-install 4.11.0-0.okd-2022-12-02-145640 built from commit c258fd3a0027ef3fa4ed685e093e044b1d4014cd release image ***@***.***:924958ed5ba93ea9d72e4b5a319d0d4df220edbb891816c106215cbf318a0a40 release architecture amd64 I also have the same problem with okd4-control-plane-1 [image: image] <https://user-images.githubusercontent.com/121056246/208499909-2d36937e-3064-4d88-a9d8-5f5284d8ada4.png> DHCP and DNS looks good to me. IP addresses and names are correct. Did you fix your issue gonhidalgo? — Reply to this email directly, view it on GitHub <#696 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRX4JNV3KQWJT36Z5LEPULWOCWMVANCNFSM463WUYGA> . You are receiving this because you were mentioned.

0 replies

cvsraman · 2024-04-12T01:18:49Z

cvsraman
Apr 12, 2024

Hi, i am trying to instll okd 4.13 snc on my Esxi 8. and tryed same on my high end laptop also.
while installing unable to start bootstarp. it is asking user id and password. "virsh console okd4-snc-bootstrap"

trying to run "virsh console okd4-snc-master", it is unable to get api server.

please help me.

0 replies

UPI/Bare Metal - The master node cannot finish the installation ("config/master", "timeout awaiting response headers"/"EOF") #696

Replies: 14 comments · 12 replies

eduardolucioac Jun 17, 2021 Author

vrutkovs Jun 17, 2021 Maintainer

eduardolucioac Jun 17, 2021 Author

vrutkovs Jun 17, 2021 Maintainer

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jan 12, 2022 Author

eduardolucioac Jun 15, 2022 Author

eduardolucioac Jun 17, 2021 Author

eduardolucioac Jun 18, 2021 Author

eduardolucioac Dec 19, 2022 Author

Replies: 14 comments 12 replies

eduardolucioac
Jun 17, 2021
Author

vrutkovs Jun 17, 2021
Maintainer

eduardolucioac Jun 17, 2021
Author

vrutkovs Jun 17, 2021
Maintainer

eduardolucioac Jun 17, 2021
Author

eduardolucioac
Jun 17, 2021
Author

eduardolucioac
Jun 17, 2021
Author

eduardolucioac
Jun 17, 2021
Author

eduardolucioac
Jun 17, 2021
Author

eduardolucioac
Jun 17, 2021
Author

eduardolucioac
Jun 17, 2021
Author

eduardolucioac
Jun 17, 2021
Author

eduardolucioac Jan 12, 2022
Author

eduardolucioac Jun 15, 2022
Author

eduardolucioac
Jun 17, 2021
Author

eduardolucioac
Jun 18, 2021
Author

eduardolucioac
Dec 19, 2022
Author