netshare volume plugin not working in "swarm mode" v1.12.0-rc3 with docker volume create, docker service create --mount

[tiagoalves83]

Output of docker version:

Client:
 Version:      1.12.0-rc3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   91e29e8
 Built:        Sat Jul  2 00:17:11 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.0-rc3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   91e29e8
 Built:        Sat Jul  2 00:17:11 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.0-rc3
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local nfs
 Network: overlay bridge host null
Swarm: active
 NodeID: 91wk9arjzrqnkzdvigu4wzcii
 IsManager: Yes
 Managers: 1
 Nodes: 4
 CACertHash: sha256:8e03c7e964a135ab6200346e7de585139e08b725d4fe62fd7dc0558104b838a6
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.14-boot2docker
Operating System: Boot2Docker 1.12.0-rc3 (TCL 7.1); HEAD : 8d9ee9f - Sat Jul  2 05:02:44 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 492.7 MiB
Name: swmaster-1
ID: I5SS:M4L6:ZKWZ:W37G:FJ24:VLMF:BCHA:LRAF:GVQ3:C2MR:INFO:WE3Y
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 35
 Goroutines: 144
 System Time: 2016-07-09T18:57:45.749715718Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8

Additional environment details:

docker-machine create -d virtualbox \
      --virtualbox-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.12.0-rc3/boot2docker.iso \
      --virtualbox-memory "512" swmaster-1

docker-machine create -d virtualbox \
      --virtualbox-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.12.0-rc3/boot2docker.iso \
      --virtualbox-memory "512" swnode-1

docker $(docker-machine config swmaster-1) swarm init --listen-addr $(docker-machine ip swmaster-1):2377

docker $(docker-machine config swnode-1) swarm join $(docker-machine ip swmaster-1):2377 --listen-addr $(docker-machine ip swnode-1):2377

netshare is running in every host, including master https://github.com/ContainX/docker-volume-netshare/releases/download/v0.18/docker-volume-netshare_0.18_linux_amd64-bin:

sudo /var/lib/boot2docker/bin/docker-volume-netshare nfs -v 3 --basedir=/mnt

netshare is working with docker run -v

docker run -it --volume-driver=nfs -v 192.168.99.1/Volumes/HDD/tmp:/mnt --name test1 alpine /bin/ls /mnt
## List all files in 192.168.99.1/Volumes/HDD/tmp OK

netshare is not working with docker volume create + docker service create

docker volume create -d nfs --name /192.168.99.1/Volumes/HDD/tmp
docker service create --endpoint-mode dnsrr --mount type=volume,source=192.168.99.1/Volumes/HDD/tmp,target=/mnt,volume-driver=nfs --name test10 alpine ping uol.com.br

Nothing is mounted ...
docker service inspect test10 shows:

"Mounts": [
                        {
                            "Type": "volume",
                            "Source": "192.168.99.1/Volumes/HDD/tmp",
                            "Target": "/mnt",
                            "VolumeOptions": {
                                "DriverConfig": {
                                    "Name": "nfs"
                                }
                            }
                        }
                    ]

docker service tasks test10 shows:

ID                         NAME      SERVICE  IMAGE   LAST STATE          DESIRED STATE  NODE
3li5cdzauffoqhrlktpu72qhx  test10.1  test10   alpine  Running 20 minutes  Running        swnode-1

Inside swnode-1 docker ps -a shows:

463fd2f94510        alpine:latest       "ping uol.com.br"   7 minutes ago       Up 7 minutes                            test10.1.3li5cdzauffoqhrlktpu72qhx

And finally, docker inspect 463fd2f94510 shows:

Mounts": [
            {
                "Name": "48fc07f3be24cc706598e3082316cb547fdd93d24df3b97c78b6c8f4862edc52",
                "Source": "/mnt/sda1/var/lib/docker/volumes/48fc07f3be24cc706598e3082316cb547fdd93d24df3b97c78b6c8f4862edc52/_data",
                "Destination": "/mnt:ro,nocopy",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],

Inside the container test10.1 a folder named /mnt:ro,nocopy is created.
$ docker exec 463fd2f94510 /bin/ls /

bin
dev
etc
home
lib
linuxrc
media
mnt
mnt:ro,nocopy
proc
root
run
sbin
srv
sys
tmp
usr
var

If I change the target= to any other folder, I still getting 'folder_name:ro,nocopy' created inside the container.

[holgerreif]

netshare is not working with docker volume create + docker service create

docker volume create -d nfs --name /192.168.99.1/Volumes/HDD/tmp

What is the output of docker volume ls afterwards?

Could you try

docker volume create -d nfs --name somename -o share=192.168.99.1/Volumes/HDD/tmp

[holgerreif]

Havn't had time to play with 1.12-rc yet, however I found the following new example in the 1.12 docs

Could you try the follwoing example from the doc for creation of the volume?

$ docker volume create --driver local --opt type=nfs --opt o=addr=192.168.1.1,rw --opt device=:/path/to/dir --name foo

But I'd say that there is a problem within the new docker service command, because it seems to change the driver from nfs to local (according to docker inspect).

Further the following output from docker service inspect looks strange:

                            "VolumeOptions": {
                                "DriverConfig": {
                                    "Name": "nfs"
                                }

I wouldn't expect to see "Name": "nfs" rather I'd expect maybe "Type": "nfs". But this is just guesing...

BTW: where did you found a detailed reference to the --mountOption of docker service create?

[tiagoalves83]

Hi, It seems docker team have done some work in 1.12.0-rc4 #24484
I will give it a try and let you know.
Do you know if ContainX is working with the experimental "docker plugin" command ?

For the --mount docs, I found this issue #24277
There is another issue (open or closed, i dont know) at https://github.com/docker/docker

[docwhat]

With docker v1.12.1 in swarm mode I get:

$ docker volume create --driver=local --opt=type=nfs --opt=o=addr=192.168.1.1,rw --opt device=:/vol/vol_docker_registry_mirror --name=registry-mirror
registry-mirror
$ docker volume ls
DRIVER              VOLUME NAME
local               registry-mirror
$ docker volume inspect registry-mirror
[
    {
        "Name": "registry-mirror",
        "Driver": "local",
        "Mountpoint": "/var/lib/docker/volumes/registry-mirror/_data",
        "Labels": {},
        "Scope": "local"
    }
]

...at which point docker run -it --rm -v registry-mirror:/data ubuntu /bin/bash does work for me.

However, I'm mildly concerned. When I ran this the first time, additional volumes (previously invisible) suddenly appeared. The volumes were my previous attempts to get this to work. :-/

[docwhat]

I was able to get a service to work, using that volume command. But I had to run the volume command by hand on every node in the swarm. It looks like it tried to auto-copy the volumes over, but it failed to work. :-/

[docwhat]

The service command (for posterity):

$ docker service create --replicas=3 --mount=type=volume,src=registry-mirror,dst=/var/lib/registry --name=mirror -p 5010:443 registry:2 serve /var/lib/registry/config.yml

[simonwahlgren]

@docwhat I can confirm that it works with your solution.

Docker version 1.12.1, build f1e1b83
Core-OS version 1164.1.0

# first create the volume on all nodes where the container is to be ran (core-03 in this example)
#
# if I don't create the volumes on each node the service will start without any error messages but the volume won't be mounted inside the container
docker volume create --driver=local --opt=type=nfs --opt=o=addr=10.10.10.66,rw --opt device=:/storage/mysql --name=mysql-staging

# start the service on core-03
docker service create --mount=type=volume,src=mysql-staging,dst=/mysql --name=nginx --constraint 'node.hostname == core-03' -p 8080:80 nginx

# verify that the volume is mounted in the container ran on core-03
docker exec a5f4b52bece4 mount | grep storage
:/storage/mysql on /mysql type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.10.66,mountvers=3,mountproto=tcp,local_lock=none,addr=10.10.10.66)

docker exec a5f4b52bece4 ls /mysql/data
auto.cnf
ib_buffer_pool
ib_logfile0
ib_logfile1
ibdata1
ibtmp1
mysql
performance_schema
sys

Everything works without docker-volume-netshare even beeing started. The only thing not working without the daemon started is docker volume ls, I get this error message:

list nfs: Post http://%2Frun%2Fdocker%2Fplugins%2Fnfs.sock/VolumeDriver.List: dial unix /run/docker/plugins/nfs.sock: connect: connection refused
DRIVER              VOLUME NAME
local               mysql-staging

But everything else is as said working.

EDIT:
After removing the file /run/docker/plugins/nfs.sock everything works.

So the question now is, why do I need docker-volume-netshare when everything seems to work out of the box?

[manast]

Could someone clarify if it is required to create the volumes on every node in the swarm in order to get this to work? currently I am getting nfs volumes in the manager, but on the rest of the nodes I just get driver=local, seems like the swarm is distributing the volumes across the nodes but does not understand that the volumes are using the netshare plugin. Furthermore, @simonwahlgren seems to manage to mount nfs using the local driver, is there any disadvantage using this approach? does it also require to create the volumes in every node?

[o3o3o]

In Docker version 1.12.3, build 6b644ec and docker-volume-netshare_0.20:

You must create nfs volume on every node, otherwise the volume will be empty.

leader,node2,node3

#run only in leader
docker volume create -d nfs --name nfshare -o share=${NFSHARE_SERVER}:/share
docker service create  \
  --mount type=volume,src=nfshare,dst=/share  \
  --publish 8080:80 \
   --replicas=10 \
   --name nginx nginx          

Then ssh to node2, node3 and enter nginx, the /share is empty.

[manast]

@o3o3o As far as I know (I learnt this recently), this is the way volumes are supposed to work in swarm. The only supported way is to create the volume when creating the service: https://docs.docker.com/engine/reference/commandline/service_create/

In my opinion not a good approach, but I hope this is just because swarm node is quite new and they have not figured out everything just yet.

[AshMartian]

Are volumes suppose to persist across docker-volume-netshare service restarts?

I setup my volumes with the docker volumes command, but when restarting the docker-volume-netshare service they disappear. This worries me that the containers will lose their volumes if the host reboots.

[manast]

@blandman they will not persist.

[AshMartian]

There's absolutely no way to define the volume when creating the service? I can get single nodes to work with the volumes, but not all nodes at the same time.

[ometa]

@blandman that's my predicament, too. I don't want to have to manually create the volume on each host.

[rdxmb]

same problem here:

root@docker3:/opt# docker volume create --driver nfs --name netsharenfs -o share=192.168.253.100:/
netsharenfs
root@docker3:/opt# docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
0zizhqemxr3fmhr949qqg94ly     docker1             Ready               Active              
88qpfbyudpeqzrfabvhptnz36     docker2             Ready               Active              
wx1fj0df7mdkihr5855xlfdcx *   docker3             Ready               Active              Leader
root@docker3:/opt# docker service create --mode=global --mount src=netsharenfs,dst=/nfs --name nfstest phusion/baseimage
fup2srz49c7p6uw9vu0bf8w82
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
root@docker3:/opt# docker volume inspect netsharenfs
[
    {
        "Driver": "nfs",
        "Labels": {},
        "Mountpoint": "/var/lib/docker-volumes/netshare/nfs/netsharenfs",
        "Name": "netsharenfs",
        "Options": {
            "share": "192.168.253.100:/"
        },
        "Scope": "global"
    }
]

root@docker1:/opt# docker volume inspect netsharenfs
[
    {
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/netsharenfs/_data",
        "Name": "netsharenfs",
        "Options": {},
        "Scope": "local"
    }
]

root@docker2:/opt# docker volume inspect netsharenfs
[
    {
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/netsharenfs/_data",
        "Name": "netsharenfs",
        "Options": {},
        "Scope": "local"
    }
]

root@docker1:/opt# service docker-volume-netshare status
● docker-volume-netshare.service - LSB: Init for docker-volume-netshare
   Loaded: loaded (/etc/init.d/docker-volume-netshare; bad; vendor preset: enabled)
   Active: active (running) since Mon 2017-08-28 13:10:35 CEST; 1h 18min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6607 ExecStart=/etc/init.d/docker-volume-netshare start (code=exited, status=0/SUCCESS)
    Tasks: 5
   Memory: 3.2M
      CPU: 55.421s
   CGroup: /system.slice/docker-volume-netshare.service
           └─6618 /usr/bin/docker-volume-netshare nfs

Aug 28 13:10:35 docker1 systemd[1]: Starting LSB: Init for docker-volume-netshare...
Aug 28 13:10:35 docker1 docker-volume-netshare[6607]:  * Starting Docker-Volume-Netshare: docker-volume-netshare
Aug 28 13:10:35 docker1 docker-volume-netshare[6607]:    ...done.
Aug 28 13:10:35 docker1 systemd[1]: Started LSB: Init for docker-volume-netshare.
Aug 28 13:11:46 docker1 systemd[1]: Started LSB: Init for docker-volume-netshare.

So nothing but the Name of the volume is replicated to the other worker nodes.
I've just had the same problem with local driver and type NFS here: moby/moby#34556

Unfortunately, the netshare-driver doesn't make this even besser.

If I can help with additional infos or tests, please contact me. I would like to help making this "swarm-ready".

//edited:
However, as swarms are mostly deployed via the yml-Files, the Volumes WITH the parameters get running on all nodes.
@simonwahlgren So the difference seems that the local driver uses NFSv3 and this here NFSv4 (by default)

[wem]

Looks to me thats this is maybe a general Swarm problem, because for me even original local volumes lost there configuration when they get applied to worker nodes

[goncalo-oliveira]

This problem has some time, so most likely won't be able to get much help, but nonetheless, I'll give it a shot, maybe someone can help...

I'm trying to use the plugin with Docker Swarm. Naturally, I don't want to have to manually create the volumes on every node, that's unthinkable, so I'm using a named volume, which means that if it doesn't exist, it's created, if it does, it re-uses it.

The command that I'm using to create the service is as follows

docker service create --name nuget-server --with-registry-auth \
--log-driver json-file --log-opt max-size=2mb \
--constraint 'node.role==worker' \
--mount 'type=volume,src=baget,dst=/var/baget,volume-driver=nfs,volume-opt=share=1.1.1.1:/mnt/volume_3/nfs/baget' \
--env ApiKey=some-random-key \
--env Storage__Type=FileSystem \
--env Storage__Path=/var/baget/packages \
--env Database__Type=Sqlite \
--env Search__Type=Database \
baget:latest

That seems to work, as it follows the parameters as when manually creating the volume. And it looks ok as well.

[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "nfs",
        "Labels": {},
        "Mountpoint": "/var/lib/docker-volumes/netshare/nfs/baget",
        "Name": "baget",
        "Options": {
            "share": "1.1.1.1:/mnt/volume_3/nfs/baget"
        },
        "Scope": "local"
    }
]

So, it all looked good, until I try to write to the folder mapped to the volume, where I get an unauthorized error. Can anyone shed some light on this? Maybe I'm not doing the volume creation properly via service create?

Cheers