Skip to content
This repository has been archived by the owner on May 31, 2024. It is now read-only.

Rqlited does not start in docker swarm mode #9

Open
sginer opened this issue Oct 9, 2019 · 8 comments
Open

Rqlited does not start in docker swarm mode #9

sginer opened this issue Oct 9, 2019 · 8 comments

Comments

@sginer
Copy link

sginer commented Oct 9, 2019

Rqlited service does not start in docker service.

When I use the docker definition of the 4.5.0 version with my own docker-entrypoint.sh with this content

#!/bin/bash

if [ "x" == "x${JOIN_ADRESS}" ]; then
        exec rqlited -http-addr ${HOSTNAME}:4001  /rqlite/file/data
else
        exec rqlited -http-addr ${HOSTNAME}:4001  -join http://${JOIN_ADRESS}:4001  /rqlite/file/data
fi

and when I launch it with this compose file whith the cmd docker stack deploy -c docker-compose.yml rqlite 

version: "3.3"

networks:
  rqlite:

services:
 rqlite1:
    image: sginer/rqlite:4.5.0
    environment:
      NODE_ID: "1"
      HOSTNAME: "rqlite1"
    networks:
      - rqlite

  rqlite2:
    image: sginer/rqlite:4.5.0
    environment:
      NODE_ID: "2"
      HOSTNAME: "rqlite2"
      JOIN_ADRESS: "rqlite1"
    networks:
      - rqlite

 rqlite3:
    image: sginer/rqlite:4.5.0
    environment:
      NODE_ID: "3"
      HOSTNAME: "rqlite3"
      JOIN_ADRESS: "rqlite1"
    networks:
      - rqlite

rqlite is unable to elect leader and to start !
@otoolep
Copy link
Member

otoolep commented Oct 9, 2019

Can you supply any logs, from the nodes? Each node should be printing a bunch of output on startup.

@sginer
Copy link
Author

sginer commented Oct 10, 2019 •
edited
Loading

Docker version

Client:
Version: 18.09.4
API version: 1.39
Go version: go1.10.8
Git commit: d14af54266
Built: Wed Mar 27 18:35:44 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 18.09.4
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: d14af54
Built: Wed Mar 27 18:01:48 2019
OS/Arch: linux/amd64
Experimental: false

Logs of master

rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:07 rqlited starting, version v4.5.0, commit 8336150318dfb2b1f196f6a4919041b65071f3fd, branch 4.3.0-patch
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:07 go1.10, target architecture is amd64, operating system target is linux
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |             _ _ _
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |            | (_) |
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |   _ __ __ _| |_| |_ ___
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |  | '__/ _  | | | __/ _ \   The lightweight, distributed
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |  | | | (_| | | | ||  __/   relational database.
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |  |_|  \__, |_|_|\__\___|
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |          | |
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    |          |_|
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [store] 2019/10/09 14:45:07 ensuring /rqlite/file/data exists
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [mux] 2019/10/09 14:45:07 mux serving on 127.0.0.1:4002, advertising 127.0.0.1:4002
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [store] 2019/10/09 14:45:07 SQLite in-memory database opened
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [store] 2019/10/09 14:45:07 enabling single-node mode
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [store] 2019/10/09 14:45:07 waiting for up to 2m0s for application of initial logs
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:07 [INFO] raft: Node at 127.0.0.1:4002 [Follower] entering Follower state (Leader: "")
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [cluster] 2019/10/09 14:45:07 service listening on 127.0.0.1:4002
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:07 no join addresses set
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:08 failed to set peer for 127.0.0.1:4002 to rqlite1:4001: no leader available (retrying)
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [WARN] raft: Heartbeat timeout from "" reached, starting election
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [INFO] raft: Node at 127.0.0.1:4002 [Candidate] entering Candidate state
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [DEBUG] raft: Votes needed: 1
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [DEBUG] raft: Vote granted from 127.0.0.1:4002. Tally: 1
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [INFO] raft: Election won. Tally: 1
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [INFO] raft: Node at 127.0.0.1:4002 [Leader] entering Leader state
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | 2019/10/09 14:45:08 [DEBUG] raft: Node 127.0.0.1:4002 updated peer set (2): [127.0.0.1:4002]
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:09 set peer for 127.0.0.1:4002 to rqlite1:4001
rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox    | [rqlited] 2019/10/09 14:45:09 failed to start HTTP server: listen tcp 10.0.7.2:4001: bind: cannot assign requested address

Logs of followers

rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | 
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |             _ _ _
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |            | (_) |
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |   _ __ __ _| |_| |_ ___
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |  | '__/ _  | | | __/ _ \   The lightweight, distributed
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |  | | | (_| | | | ||  __/   relational database.
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |  |_|  \__, |_|_|\__\___|
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |          | |
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    |          |_|
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | 
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:26 rqlited starting, version v4.5.0, commit 8336150318dfb2b1f196f6a4919041b65071f3fd, branch 4.3.0-patch
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:26 go1.10, target architecture is amd64, operating system target is linux
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:26 ensuring /rqlite/file/data exists
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [mux] 2019/10/10 06:15:26 mux serving on 127.0.0.1:4002, advertising 127.0.0.1:4002
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:26 SQLite in-memory database opened
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:26 waiting for up to 2m0s for application of initial logs
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | 2019/10/10 06:15:26 [INFO] raft: Node at 127.0.0.1:4002 [Follower] entering Follower state (Leader: "")
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster] 2019/10/10 06:15:26 service listening on 127.0.0.1:4002
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:26 join addresses are: [http://rqlite1:4001]
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster-join] 2019/10/10 06:15:26 failed to join cluster at [http://rqlite1:4001], sleeping 5s before retry
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | 2019/10/10 06:15:27 [WARN] raft: EnableSingleNode disabled, and no known peers. Aborting election.
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster-join] 2019/10/10 06:15:31 failed to join cluster at [http://rqlite1:4001], sleeping 5s before retry
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster-join] 2019/10/10 06:15:36 failed to join cluster at [http://rqlite1:4001], sleeping 5s before retry
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [cluster-join] 2019/10/10 06:15:41 failed to join cluster at [http://rqlite1:4001], after 3 attempts
rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:41 failed to join cluster at [http://rqlite1:4001]: Post http://rqlite1:4001/join: dial tcp 10.0.8.5:4001: connect: connection refused
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:49 rqlited starting, version v4.5.0, commit 8336150318dfb2b1f196f6a4919041b65071f3fd, branch 4.3.0-patch
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [rqlited] 2019/10/10 06:15:49 go1.10, target architecture is amd64, operating system target is linux
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:49 ensuring /rqlite/file/data exists
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [store] 2019/10/10 06:15:49 SQLite in-memory database opened
rqlite_rqlite2.1.6c4x4bmtu00p@ubuntu-VirtualBox    | [mux] 2019/10/10 06:15:49 mux serving on 127.0.0.1:4002, advertising 127.0.0.1:4002

@sginer sginer changed the title Rqlited does not start in docker swar mode Rqlited does not start in docker swarm mode Oct 10, 2019
@otoolep
Copy link
Member

otoolep commented Oct 10, 2019

Something is wrong with your network setup:

rqlite_rqlite1.1.wz6k678mq9pj@ubuntu-VirtualBox | [rqlited] 2019/10/09 14:45:09 failed to start HTTP server: listen tcp 10.0.7.2:4001: bind: cannot assign requested address

@sginer
Copy link
Author

sginer commented Oct 10, 2019 •
edited
Loading

I'm agree with you. Is raft use a spécific protocol or need a specific network configuration (like multicast or other). Have you ever test rqlite in a swarm environnement ? When I use it on a single node with docker compose all work fine.

@otoolep
Copy link
Member

otoolep commented Oct 10, 2019

I have not tested it in a Swarm environment. An rqlite node opens up two TCP ports -- one for internode Raft communication, and the other for the HTTP server.

I suggest that either that address/port is in use, or is not even visible to the rqlite node.

@otoolep
Copy link
Member

otoolep commented Oct 10, 2019

This is also a sign of problems: rqlite_rqlite2.1.i7x16w8y7hcv@ubuntu-VirtualBox | [cluster-join] 2019/10/10 06:15:26 failed to join cluster at [http://rqlite1:4001], sleeping 5s before retry

@otoolep
Copy link
Member

otoolep commented Oct 10, 2019

Perhaps you should mail the rqlite Google Group, someone there may be able to help.

@mikeTWC1984
Copy link

After struggling for a while I got it to work using official image:

version: '3.8'

services:
  rqlite1:
    hostname: 'RQX{{.Task.Slot}}'
    image: rqlite/rqlite
    command: [ "-join=RQX1:4002,RQX2:4002,RQX3:4002", "-bootstrap-expect=3" ]
    deploy:
      replicas: 3
    ports:
      - "4001:4001"
    #   - "4002:4002"
    networks:
      - main
networks:
  main:
    external: true

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@otoolep @sginer @mikeTWC1984