KeepaliveD On Startup Not Accepting Connections

[ArcticSnowman]

When the keepalived-cip initial starts the master attempt to setup the routes, but for some reason it wont accept connections until a change in the targets, like a scaling of echoheaders, causes it to re-load the keepalived config...

I0904 18:29:22.083291       1 main.go:231] cleaning ipvs configuration
I0904 18:29:22.096630       1 main.go:185] Creating API server client for https://172.18.11.1:443
I0904 18:29:22.097312       1 main.go:133] starting LVS configuration
I0904 18:29:22.123116       1 main.go:372] No interface was provided, proceeding with the node's default: eth0
W0904 18:29:22.151516       1 main.go:90] requeuing default/kubernetes, err unexpected error searching configmap keepalived-vip/ops-keepalive-kube-keepalived-vip: configmap ops-keepalive-kube-keepalived-vip was not found
I0904 18:29:22.236407       1 main.go:304] starting keepalived to announce VIPs
Tue Sep  4 18:29:22 2018: Starting Keepalived v1.4.4 (unknown)
Tue Sep  4 18:29:22 2018: WARNING - keepalived was build for newer Linux 4.9.88, running on Linux 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018
Tue Sep  4 18:29:22 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:29:22 2018: Starting Healthcheck child process, pid=20
Tue Sep  4 18:29:22 2018: Starting VRRP child process, pid=21
Tue Sep  4 18:29:22 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:29:22 2018: Registering Kernel netlink reflector
Tue Sep  4 18:29:22 2018: Registering Kernel netlink command channel
Tue Sep  4 18:29:22 2018: Registering gratuitous ARP shared channel
Tue Sep  4 18:29:22 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:29:22 2018: Using LinkWatch kernel netlink reflector...
I0904 18:29:24.154581       1 keepalived.go:168] reloading keepalived
Tue Sep  4 18:29:24 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:29:24 2018: Got SIGHUP, reloading checker configuration
Tue Sep  4 18:29:24 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:29:24 2018: Initializing ipvs
Tue Sep  4 18:29:24 2018: Gained quorum 1+0=1 <= 1 for VS [192.168.169.165]:tcp:80
Tue Sep  4 18:29:24 2018: Gained quorum 1+0=1 <= 1 for VS [192.168.169.205]:tcp:80
Tue Sep  4 18:29:24 2018: Gained quorum 1+0=1 <= 1 for VS [192.168.169.205]:tcp:443
Tue Sep  4 18:29:24 2018: Activating healthchecker for service [172.21.11.21]:tcp:8080 for VS [192.168.169.165]:tcp:80
Tue Sep  4 18:29:24 2018: Activating healthchecker for service [172.21.11.176]:tcp:80 for VS [192.168.169.205]:tcp:80
Tue Sep  4 18:29:24 2018: Activating healthchecker for service [172.21.11.176]:tcp:443 for VS [192.168.169.205]:tcp:443
Tue Sep  4 18:29:24 2018: Registering Kernel netlink reflector
Tue Sep  4 18:29:24 2018: Registering Kernel netlink command channel
Tue Sep  4 18:29:24 2018: Registering gratuitous ARP shared channel
Tue Sep  4 18:29:24 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:29:24 2018: Using LinkWatch kernel netlink reflector...
Tue Sep  4 18:29:24 2018: VRRP_Instance(vips) Entering BACKUP STATE
Tue Sep  4 18:29:24 2018: VRRP_Instance(vips) removing protocol VIPs.
Tue Sep  4 18:29:24 2018: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Tue Sep  4 18:29:27 2018: VRRP_Instance(vips) Transition to MASTER STATE
Tue Sep  4 18:29:28 2018: VRRP_Instance(vips) Entering MASTER STATE
Tue Sep  4 18:29:28 2018: VRRP_Instance(vips) using locally configured advertisement interval (1000 milli-sec)
Tue Sep  4 18:29:28 2018: VRRP_Instance(vips) setting protocol VIPs.
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:28 2018: VRRP_Instance(vips) Sending/queueing gratuitous ARPs on eth0 for 192.168.169.165
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:28 2018: VRRP_Instance(vips) Sending/queueing gratuitous ARPs on eth0 for 192.168.169.205
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:28 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:33 2018: VRRP_Instance(vips) Sending/queueing gratuitous ARPs on eth0 for 192.168.169.165
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:33 2018: VRRP_Instance(vips) Sending/queueing gratuitous ARPs on eth0 for 192.168.169.205
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.165
Tue Sep  4 18:29:33 2018: Sending gratuitous ARP on eth0 for 192.168.169.205
I0904 18:30:40.152106       1 keepalived.go:168] reloading keepalived
Tue Sep  4 18:30:40 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:30:40 2018: Got SIGHUP, reloading checker configuration
Tue Sep  4 18:30:40 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:30:40 2018: Initializing ipvs
Tue Sep  4 18:30:40 2018: Activating healthchecker for service [172.21.11.145]:tcp:8080 for VS [192.168.169.165]:tcp:80
Tue Sep  4 18:30:40 2018: Activating healthchecker for service [172.21.11.21]:tcp:8080 for VS [192.168.169.165]:tcp:80
Tue Sep  4 18:30:40 2018: Activating healthchecker for service [172.21.11.176]:tcp:80 for VS [192.168.169.205]:tcp:80
Tue Sep  4 18:30:40 2018: Activating healthchecker for service [172.21.11.176]:tcp:443 for VS [192.168.169.205]:tcp:443
Tue Sep  4 18:30:40 2018: Registering Kernel netlink reflector
Tue Sep  4 18:30:40 2018: Registering Kernel netlink command channel
Tue Sep  4 18:30:40 2018: Registering gratuitous ARP shared channel
Tue Sep  4 18:30:40 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Sep  4 18:30:40 2018: VRRP_Instance(vips) setting protocol VIPs.
Tue Sep  4 18:30:40 2018: Using LinkWatch kernel netlink reflector...
Tue Sep  4 18:30:40 2018: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

This is deployed with the helm chart.

[steven-sheehy]

Not sure what you mean. It looks to me like it starts at 18:29:22.083291 and then two seconds later at 18:29:24.154581 it reloads the keepalived configuration and begins sending gratuitous ARP for the new VIPs. The two second gap is because it starts keepalived with an empty configuration until it can retrieve the keepalived configmap and service endpoints to populate it with the actual configuration. Are you saying this 2 second gap is an issue?

[ArcticSnowman]

The keepalived does not accept connections until I manually triggered an update in the config..

[steven-sheehy]

Please try with latest version.