Blockchain error after restarting kubernetes nodes

[deanzaka]

I was following the instruction from here https://ibm-blockchain.github.io/setup/ to setup blockchain on kubernetes cluster. The setup was successful.

But, after I tried to reboot the node in kubernetes cluster, the orderer failed to restart, as shown below:

[mrshah-at-ibm]

@deanzaka can you let us know if you are using free cluster on IBM Container Service? Looks like the persistent storage where the bootstrap files for the orderer are stored didn't survive the reboot. We can replay the scenario and let you know how it goes.

[deanzaka]

@mrshah-at-ibm yes, we are using free cluster on IBM Container Service.
Thanks for your help, please let us know if there are any updates.

[deanzaka]

hi @mrshah-at-ibm, do you have any update on this issue?

[mrshah-at-ibm]

I do not have any update yet. We will give it a try this week. Sorry for the delay.

[deanzaka]

Yea, no worries. Thanks a lot guys.

[mrshah-at-ibm]

We found one of the issues (IBM-Blockchain-Archive/ibm-container-service#61) but it might not be the one causing this. We will still need to continue testing.

[deanzaka]

Oh, yea, I actually notice that too. If we delete the pods then the kube deployments create it again, after that we still can query the data but can't invoke any transaction because we lost world state and the business network somehow. I wasn't sure if it's an expected behavior, but I leave it for now.

It'll be great if we can solve both of those problems in one solution.
I'll keep troubleshooting this issue from my side too. Thanks guys.

[mrshah-at-ibm]

@deanzaka I have not been able to replay the scenario, but afaik the "Reload" will reload the OS on the worker node, which will wipe the locally mounted file system data folders.. And thus, the orderer will not have stuff to start - this stuff is generated by the utils pod.

I hope this helps.

[dhyey20]

@deanzaka based on some tests that we ran, if the pod gets restarted (i.e. either you scale the deployment down to 0 and then up to 1, or somehow restart the container) the peer/orderer lose the ledger data, as mentioned in issue. But there is no error.

But the case where someone tries to restart the kubernetes-worker node, using bx cs worker-reboot <cluster-name> <worker-name> in free cluster, the storage gets wiped out. So after restart the /shared folder is blank. This is because we are mounting the /tmp folder on the host to /shared on the containers. Investigating further to see if this can be solved (tracked here).

[dhyey20]

This issue has been fixed with IBM-Blockchain-Archive/ibm-container-service#67. Feel free to reopen if you still see problems.

(please pull the latest code and try again)