Documenting solutions to common issues faced while restarting glusterd2 service #885

rishubhjain · 2018-06-13T21:23:17Z

Issue: #883

rishubhjain · 2018-06-13T21:36:42Z

@atinmu I have just documented solutions to some of the issues faced while restarting glusterd2 service. I would like to know if this is a suitable way of documenting the solutions to the common issues?

Madhu-1 · 2018-06-14T09:52:59Z

doc/quick-start-user-guide.md

+  The path to default directory used by glusterd2 is ["/var/lib/glusterd/"](https://github.com/gluster/glusterd2/blob/master/glusterd2.toml.example#L1), if using custom config file then please provide working directory path instead of "/var/lib/glusterd/"
+
+  ```sh
+  # rm /var/lib/glusterd/*


do we need to delete the complete glusterd2 directory? it will delete the old ETCD data which is stored in ETCD folder. ETCD config will be stored in store.toml (I think clearing this should be enough). If store.toml is empty glusterd2 should regenerate the store.toml based on the configuration

This comment is still not addressed?

@Madhu-1 I will provide a warning, and will send a new PR after testing whether its suffiecient to delete store.toml.

atinmu · 2018-06-14T13:58:24Z

@atinmu I have just documented solutions to some of the issues faced while restarting glusterd2 service. I would like to know if this is a suitable way of documenting the solutions to the common issues?

This is definitely a good start and we should make this as a continuous process to capture all the troubleshooting experience. However what I am additionally looking at the document is more of a concrete direction on "Do these x, y, z things before setting up the environment"

rishubhjain · 2018-06-23T16:55:02Z

@atinmu Is this PR good to go, or would you like to reformat the documenting?

atinmu · 2018-06-25T04:20:59Z

@atinmu Is this PR good to go, or would you like to reformat the documenting?

I don't see any reason why this PR can not get in. We need to get into a good habit of refreshing this document in a periodic basis or whenever we stumble upon some issues which an user can face frequently.

aravindavk · 2018-06-25T05:20:16Z

doc/quick-start-user-guide.md

+  ERRO[2018-06-04 06:05:41.020313] failed to start embedded store                error="dial tcp {IP}: connect: connection refused" source="[embed.go:36:store.newEmbedStore]"
+  ```
+  To solve this you will be required to clean up glusterd2 [working directory](https://github.com/gluster/glusterd2/blob/master/doc/quick-start-user-guide.md#running-glusterd2).
+  The path to default directory used by glusterd2 is ["/var/lib/glusterd/"](https://github.com/gluster/glusterd2/blob/master/glusterd2.toml.example#L1), if using custom config file then please provide working directory path instead of "/var/lib/glusterd/"


default is /var/lib/glusterd2

aravindavk · 2018-06-25T05:22:19Z

doc/quick-start-user-guide.md

+
+  Sample Output:
+  ```log
+  FATA[2018-06-04 06:07:22.605017] Failed to create pid file                     error="Process is already running" source="[main.go:87:main.main]"


This should not occur if pid file exists and process is not running. Please recheck

aravindavk · 2018-06-25T05:23:25Z

doc/quick-start-user-guide.md

+  To delete the pid file for the given output:
+
+  ```sh
+  # rm /usr/local/var/run/glusterd2/glusterd2.pid


If other instance of glusterd2 is running, removing pid file is not sufficient.

aravindavk · 2018-06-25T05:24:56Z

doc/quick-start-user-guide.md

+  ```log
+  FATA[2018-06-04 06:07:38.348333] failed to listen                              error="listen unix /usr/local/var/run/glusterd2/glusterd2.socket: bind: address already in use" socket=glusterd2.socket source="[server.go:76:sunrpc.NewMuxed]"
+  ```
+  This issue occurs because the socket address is already in use by old glusterd service. To resolve this issue you will have to delete the socket file to free the socket for the new glusterd service.


Also add a note to check if any other instance of glusterd2 is running before removing the socket file

aravindavk · 2018-06-25T05:26:35Z

doc/quick-start-user-guide.md

+  ```log
+  ERRO[2018-06-04 06:05:41.020313] failed to start embedded store                error="dial tcp {IP}: connect: connection refused" source="[embed.go:36:store.newEmbedStore]"
+  ```
+  To solve this you will be required to clean up glusterd2 [working directory](https://github.com/gluster/glusterd2/blob/master/doc/quick-start-user-guide.md#running-glusterd2).


Deleting the workdir will erase all cluster's data. Suggest the workaround only if this issue faced during re-setup.

rishubhjain · 2018-07-12T13:45:43Z

@aravindavk Please review, I have made the changes

phlogistonjohn · 2018-07-12T20:48:14Z

doc/quick-start-user-guide.md

+  The path to default directory used by glusterd2 is "/var/lib/glusterd2/", if using custom config file then please provide working directory path instead of "/var/lib/glusterd2/"
+
+  ```sh
+  # rm /var/lib/glusterd/2*


Is this path correct? Above you write "/var/lib/glusterd2/" but here the last slash and the 2 are transposed.

prashanthpai

Some of the issues aren't major and may seem obvious from symptoms and error messages. We'll have to focus on documenting cluster restart and quorum loss.

prashanthpai · 2018-07-13T11:37:52Z

doc/quick-start-user-guide.md

+  ```log
+  FATA[2018-06-04 06:07:38.348333] failed to listen                              error="listen unix /usr/local/var/run/glusterd2/glusterd2.socket: bind: address already in use" socket=glusterd2.socket source="[server.go:76:sunrpc.NewMuxed]"
+  ```
+  This issue occurs because the socket address is already in use by old glusterd service. To resolve this issue you will have to delete the socket file to free the socket for the new glusterd service.


The socket file isn't cleaned up on ungraceful or abrupt shutdown such as SIGKILL or a crash.

prashanthpai · 2018-07-13T11:40:54Z

doc/quick-start-user-guide.md

+
+### Solutions to common issues
+
+  >  Note: These are hacks or temporary solutions to the problems. Only use these solutions only if the issues are faced during re-setup.


These are hacks or temporary solutions to the problems.

Doesn't instil confidence on glusterd2, from user's perspective ;)

prashanthpai · 2018-07-13T11:42:17Z

doc/quick-start-user-guide.md

+
+  >  Note: These are hacks or temporary solutions to the problems. Only use these solutions only if the issues are faced during re-setup.
+
+* If glusterd service fails with error: "failed to start embedded store"


You don't have to start every sentence with `If glusterd service fails with error. Just the symptom should suffice.

prashanthpai · 2018-07-13T13:03:00Z

doc/quick-start-user-guide.md

+  ```log
+  ERRO[2018-06-04 06:05:41.020313] failed to start embedded store                error="dial tcp {IP}: connect: connection refused" source="[embed.go:36:store.newEmbedStore]"
+  ```
+  To solve this you will be required to clean up glusterd2 [working directory](https://github.com/gluster/glusterd2/blob/master/doc/quick-start-user-guide.md#running-glusterd2).


It's localstatedir, and not working directory.

prashanthpai · 2018-07-13T13:04:38Z

doc/quick-start-user-guide.md

+  # rm /usr/local/var/run/glusterd2/glusterd2.pid
+  ```
+
+* If glusterd service fails with error: "failed to listen"


"failed to listen" can also mean another glusterd2 process is running. You have to be a little more specific about the socket file and what caused it

prashanthpai · 2018-07-13T13:05:42Z

doc/quick-start-user-guide.md

+  ```log
+  FATA[2018-06-04 06:07:38.348333] failed to listen                              error="listen unix /usr/local/var/run/glusterd2/glusterd2.socket: bind: address already in use" socket=glusterd2.socket source="[server.go:76:sunrpc.NewMuxed]"
+  ```
+  This issue occurs because the socket address is already in use by old glusterd service. To resolve this issue you will have to delete the socket file to free the socket for the new glusterd service.


because the socket address is already in use by old glusterd service.

It's not.

Either:

It was being in use but wasn't cleaned up on shutdown

There's another glusterd2 instances running

rishubhjain changed the title ~~Documenting etcd cleanups~~ Documenting solutions to common issues faced while restarting glusterd2 service Jun 14, 2018

prashanthpai requested review from aravindavk, kshlm and Madhu-1 June 14, 2018 09:39

Madhu-1 reviewed Jun 14, 2018

View reviewed changes

aravindavk suggested changes Jun 25, 2018

View reviewed changes

atinmu added the GCS-Sprint1-review-pending label Jul 9, 2018

prashanthpai requested a review from phlogistonjohn July 12, 2018 12:18

Documenting etcd cleanups

79b85e6

rishubhjain force-pushed the doc branch from f8d99bd to 79b85e6 Compare July 12, 2018 13:06

phlogistonjohn suggested changes Jul 12, 2018

View reviewed changes

Merge branch 'master' of https://github.com/gluster/glusterd2 into doc

a05b8ec

rishubhjain force-pushed the doc branch from 4c9d97f to a05b8ec Compare July 13, 2018 01:52

Merge branch 'master' of https://github.com/gluster/glusterd2 into doc

992c0c2

prashanthpai suggested changes Jul 13, 2018

View reviewed changes

prashanthpai added the GCS-Sprint2-review-pending label Jul 16, 2018

atinmu removed GCS-Sprint1-review-pending labels Oct 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Documenting solutions to common issues faced while restarting glusterd2 service #885

Documenting solutions to common issues faced while restarting glusterd2 service #885

rishubhjain commented Jun 13, 2018

rishubhjain commented Jun 13, 2018

Madhu-1 Jun 14, 2018

atinmu Jun 25, 2018

rishubhjain Jul 12, 2018

atinmu commented Jun 14, 2018

rishubhjain commented Jun 23, 2018

atinmu commented Jun 25, 2018

aravindavk Jun 25, 2018

aravindavk Jun 25, 2018

aravindavk Jun 25, 2018

aravindavk Jun 25, 2018

aravindavk Jun 25, 2018

rishubhjain commented Jul 12, 2018

phlogistonjohn Jul 12, 2018

prashanthpai left a comment

prashanthpai Jul 13, 2018

prashanthpai Jul 13, 2018

prashanthpai Jul 13, 2018

prashanthpai Jul 13, 2018

prashanthpai Jul 13, 2018

prashanthpai Jul 13, 2018


		### Solutions to common issues

		> Note: These are hacks or temporary solutions to the problems. Only use these solutions only if the issues are faced during re-setup.


		> Note: These are hacks or temporary solutions to the problems. Only use these solutions only if the issues are faced during re-setup.

		* If glusterd service fails with error: "failed to start embedded store"

Documenting solutions to common issues faced while restarting glusterd2 service #885

Are you sure you want to change the base?

Documenting solutions to common issues faced while restarting glusterd2 service #885

Conversation

rishubhjain commented Jun 13, 2018

rishubhjain commented Jun 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atinmu commented Jun 14, 2018

rishubhjain commented Jun 23, 2018

atinmu commented Jun 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rishubhjain commented Jul 12, 2018

Choose a reason for hiding this comment

prashanthpai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment