Zookeeper-operator does not restart pods when static configs are changed

[tylergu]

Description

When I change the fields under spec.config, zookeeper-operator does not issue a rolling update to reflect the changed config. For example, if I change the spec.config.commitLogCount field to 100, the operator reconciles the configMap to reflect the change. So in the pod, /conf/zoo.cfg which is where the configMap mounted has the commitLogCount field set to 100. But the /data/conf/zoo.cfg which is the config used by zookeeper still has the commitLogCount set to default value as 500.

Steps to reproduce:

1. Deploy a simple zookeeper cluster with the following yaml:

apiVersion: "zookeeper.pravega.io/v1beta1"
kind: "ZookeeperCluster"
metadata:
  name: "zookeeper"
spec:
  replicas: 3

2. Change the zookeeper yaml to configure the commitLogCount to 100

apiVersion: "zookeeper.pravega.io/v1beta1"
kind: "ZookeeperCluster"
metadata:
  name: "zookeeper"
spec:
  replicas: 3
  config:
    commitLogCount: 100

3. Observe that the pods didn't restart, and go into the pod and check the config used by zookeeper in /data/conf/zoo.cfg:

metricsProvider.exportJvmInfo=true
dataDir=/data
4lw.commands.whitelist=cons, envi, conf, crst, srvr, stat, mntr, ruok
syncLimit=2
commitLogCount=500
metricsProvider.httpPort=7000
snapSizeLimitInKb=4194304
standaloneEnabled=false
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
initLimit=10
minSessionTimeout=4000
snapCount=10000
admin.serverPort=8080
autopurge.purgeInterval=1
maxSessionTimeout=40000
maxCnxns=0
globalOutstandingLimit=1000
reconfigEnabled=true
skipACL=yes
autopurge.snapRetainCount=3
tickTime=2000
quorumListenOnAllIPs=false
preAllocSize=65536
maxClientCnxns=60
dynamicConfigFile=/data/conf/zoo.cfg.dynamic.100000010
root@zookeeper-0:/data/conf# cat zoo.cfg | grep con
4lw.commands.whitelist=cons, envi, conf, crst, srvr, stat, mntr, ruok
reconfigEnabled=true
dynamicConfigFile=/data/conf/zoo.cfg.dynamic.100000010

Importance

should-have

Location

Zookeeper-operator is missing the functionality to restart the pods when config is changed.

Suggestions for an improvement

We suggest to attach the hash of config as annotations to zookeeper's statefulSet's template. So that when the config is changed, the changed annotation would trigger statefulSet's rolling update.

[tylergu]

This bug is manually found by @laphets.

[tylergu]

Issued: pravega/zookeeper-operator#454

[tianyin]

This bug is manually found by @laphets.

I'm confused.

[tylergu]

@tianyin Wenqing noticed that zookeeper-operator does not restart the pods when the config is changed when he was using the zookeeper-operator. We haven't used Acto to test zookeeper-operator yet.

And our current oracle wouldn't be able to find this bug, because we don't have application specific knowledge to check whether the application used the new config. We would report false negative because there is a matching delta in the configMap.

[tylergu]

confirmed