Namenodes fail to recover after K8s cluster nodes with active namenode and zookeeper pod restart

[fanbowen13013]

If you tried to shutdown K8S node where active namenode and zookeeper pod locate, you may find the HA cluster can not recover due to some name resolving issues even after node rebooted. You have to recover the cluster manually. During that time the whole cluster is offline.

[kimoonkim]

@fanbowen13013, thanks for trying out hdfs-on-k8s and pointing out the issue. We are hoping to collect chaos test scenarios so that we can harden hdfs-on-k8s against them.

Your fail scenario is certainly interesting and we'd love to reproduce and look into it. (I saw some issues with DNS in the past myself and I was suspecting something like your scenario can happen) Do you still have access to the error messages from the namenode or zookeeper? If yes, can you please share them here? That'll allow us to see what we reproduce later will be the same errors or not.

[kimoonkim]

And sorry about your trouble. hdfs-on-k8s is currently at pre-alpha quality and certainly not production ready yet. We are hoping to reach alpha state soon with the plan in #38. We are definitely interested in testing and hardening against your fail scenario before we reach the alpha release.

[fanbowen13013]

Your project really help me a lot to build a HA HDFS cluster on K8S. I just want to remind you, in case that you haven't test my previous fail scenario.Right now, my HA HDFS can be accessed from both pods inside K8S and nodes outside. Here is my solution. I solved the problem by using hostNetwork and adding tag to nodes to deploy zookeeper clutser. I replace all the pods name in my configuration with nodes hostname. In order to resolve nodes hostname inside pod, I chose to mount /etc/hosts into kubedns instead of using hostNetwork. After that, I did some test. I shutdown the node which active namenode and zookeeper pod locate in. The standby namenode became active successfully and the whole cluster still can offer normal service.

[fanbowen13013]

Sorry, I didn't keep the error message. But I remembered that the standby namenode just crashed when I shutdown the cluster. The error log told me that the standby namenode couldn't not resolve the pod name of offline zookeeper pod. The whole cluster couldn't recover even I rebooted the node. And I also did the test to shutdown the node with standy namenode and tried to upload a file. The active namenode just malfunctioned due to failing to resolve the pod name of standy namenode. All the pod names in hdfs-site.xml should be replaced by node ips or hostnames.

[kimoonkim]

Thanks for the information, @fanbowen13013. I think that really helps us narrow down what components to look at when we reproduce your scenario.

[kimoonkim]

Ah, good to know. Thanks, Kimoon

…

On Fri, Jun 15, 2018 at 12:12 AM, fanbowen13013 ***@***.***> wrote: For the zookeeper headless service, I forget to add " service.alpha.kubernetes.io/tolerate-unready-endpoints: true" annotation. After I added this annotation, all the problem were solved — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFpBGARUbg9IDW3twewsk0ivCz7QsBI-ks5t815tgaJpZM4Tp583> .

[fanbowen13013]

I made more tests last weekend and tried to restart the node with namenode in turns. Find out standby namenode could make transition to active successfully at the first time. After serveral restarts, the cluster just brain split