Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8

Open
errordeveloper opened this issue Jan 28, 2015 · 30 comments

Comments

@errordeveloper
Copy link
Collaborator

There is an additional issue, on a number of distros /usr/bin/docker requires libdevmapper.so, as the storage backends differ.

@errordeveloper
Copy link
Collaborator Author

I have tried bind-mounting the library, but that require ldconf and it got very messy.

I also tried installing the debian version of the library, and that gave me some version, but not the one the binary wanted 😢

A static docker client-only binary would be a feasible solution, I suppose.

@errordeveloper
Copy link
Collaborator Author

This relates to #5, but the angle is quite different here...

@binocarlos
Copy link
Owner

I just hit this issue trying to get the container to work on fedora:

docker: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory

@rade
Copy link
Collaborator

rade commented Feb 27, 2015

Why are you running the full docker? You should just run a docker client. Similar to what you get on MacOS. Unfortunately, afaict, the only way to get hold of such a client-only docker exe is to build it yourself from the docker source tree.

@binocarlos
Copy link
Owner

@rade ahhh interesting - I was searching for statically linked docker binary - do you mean that there is a "client only" docker binary? this would be much better to include than the "full docker" because basically all I need it to do it HTTP requests (to the full server).

Perhaps a generic "docker source compile container" is called for that spits out full and client only versions of the docker binary. Thanks for the insight - I had not thought of anything other than "the standard docker binary" to do the client requests with...

@rade
Copy link
Collaborator

rade commented Feb 28, 2015

do you mean that there is a "client only" docker binary

Yes. I believe that's what you get on OSX, since there the docker exe always talks to a daemon that runs in a VM. (so I'm told; I'm not a Mac user).

Looking at the docker source code, there's a daemon build tag which controls whether to build an exe that includes the daemon portion. It's set during the build unless DOCKER_CLIENT_ONLY is set.

@psfales
Copy link

psfales commented Mar 4, 2015

I'm seeing the same thing here on CentOS7 (same as RedHat 7)
$ docker logs powerstrip-weave
docker: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory
docker: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory

@binocarlos
Copy link
Owner

So I've compiled a statically linked DOCKER_CLIENT_ONLY docker binary which is version 1.3.1 and included it into the image.

This means:

  • it will not require anything else (like devicemapper) because it's just a client binary
  • the parent container does not need /usr/bin/docker mounted

@rade thanks for the heads up - works beautifully - @psfales thanks for your work in diagnosing this - @errordeveloper thanks for finding it :-)

I'm going to tentatively close this issue...

@psfales
Copy link

psfales commented Apr 22, 2015

Hi Kai,

After not doing anything with powerstrip for a while, I went back to
look at it again. Now I can't make it work.

I don't know for sure sure what the problem is, but I think it may
be related to the fact that the underlying RedHat system has been
updated from docker-1.3 to docker-1.5. When I start the
powestrip-weave container, docker logs shows:

2015/04/22 20:38:14 Cannot connect to the Docker daemon. Is 'docker -d'
running on this host?
2015/04/22 20:38:14 Cannot connect to the Docker daemon. Is 'docker -d'
running on this host?
server listening on port: 80

I think this is because the 1.3 client is not able to talk to the 1.5
server using the the socket.

But I think it might be a bug with docker... When I look at
/var/run/docker.sock from inside the container, it's marked as "file"
not "socket"

Have you heard of any problems related to docker-1.5?

On Tue, Mar 10, 2015 at 09:39:01AM -0700, Kai Davenport wrote:

So I've compiled a statically linked DOCKER_CLIENT_ONLY docker binary which is version 1.3.1 and included it into the image.

This means:

  • it will not require anything else (like devicemapper) because it's just a client binary
  • the parent container does not need /usr/bin/docker mounted

@rade thanks for the heads up - works beautifully - @psfales thanks for your work in diagnosing this - @errordeveloper thanks for finding it :-)

I'm going to tentatively close this issue...


Reply to this email directly or view it on GitHub:
#8 (comment)

Peter Fales
Alcatel-Lucent
Member of Technical Staff
1960 Lucent Lane
Room: 9H-505
Naperville, IL 60566-7033
Email: [email protected]
Phone: 630 979 8031

@binocarlos binocarlos reopened this Apr 23, 2015
@binocarlos
Copy link
Owner

@psfales Hey!

Can you give me a log of the commands you used to spin up powerstrip and powerstrip-weave?

I don't think it's anything to do with docker 1.5 - I've been happily running powerstrip & powerstrip-weave against docker 1.5 for a while. If I can replicate your setup in a VM then I can track it down :-)

Thanks!

@psfales
Copy link

psfales commented Apr 23, 2015

To follow up my earlier mail... Can you try it in a CentOS7 VM? That's
where I'm seeing the problems.

Peter Fales
Peter "at" fales-lorenz.net

On Thu, Apr 23, 2015 at 04:27:04AM -0700, Kai Davenport wrote:

[1]@psfales Hey!

Can you give me a log of the commands you used to spin up powerstrip
and powerstrip-weave?

I don't think it's anything to do with docker 1.5 - I've been happily
running powerstrip & powerstrip-weave against docker 1.5 for a while.
If I can replicate your setup in a VM then I can track it down :-)

Thanks!

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@psfales
Copy link

psfales commented Apr 23, 2015

Ok, now I'm more confused.

There is definitely a problem with docker-1.5 on redhat. As I mentioned
before, it's not properly handling the volume mount of
/var/run/docker.sock, and I had to workaround it by privately installing
a 1.6-rc version.

Yesterday, even with docker-1.6, the powerstrip containers were not
starting up. But when I tried to reproduce the problem today, it worked!
I don't know of anything that I'm doing differently.

One more question... I'm seeing numerous instances of this signature
in the logs:

2015-04-23 14:59:23+0000 [DockerProxyClient,client] Stopping factory
<powerstrip.powerstrip.DockerProxyClientFactory instance at 0x7ff018a708c0>

When I saw that yesterday, I was assuming that it meant things were
failing and explained why the WEAVE_CIDR containers were not being
started. But since I'm seeing the same messages today, I guess
that's normal. Is that right?

As long as that's not a problem, I guess you can blame this on
redhat-introduced breakage, and not a problem with powerstrip-weave

Peter Fales
Peter "at" fales-lorenz.net

On Thu, Apr 23, 2015 at 04:27:04AM -0700, Kai Davenport wrote:

[1]@psfales Hey!

Can you give me a log of the commands you used to spin up powerstrip
and powerstrip-weave?

I don't think it's anything to do with docker 1.5 - I've been happily
running powerstrip & powerstrip-weave against docker 1.5 for a while.
If I can replicate your setup in a VM then I can track it down :-)

Thanks!

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@binocarlos
Copy link
Owner

@psfales The Stopping factory stuff in the logs is the Python Twisted libraries way of saying "I did a connection" - this always trips me up because it looks like something is failing - it's perfectly normal however :)

Is everything working OK with 1.6-rc?

@psfales
Copy link

psfales commented Apr 29, 2015

OK, thanks. Yes, (other than I thought that it did not work the first
time I tried 1.6) it seems to be working just fine. I have not seen
any other problems

Peter Fales
Peter "at" fales-lorenz.net

On Wed, Apr 29, 2015 at 04:50:30AM -0700, Kai Davenport wrote:

[1]@psfales The Stopping factory stuff in the logs is the Python
Twisted libraries way of saying "I did a connection" - this always
trips me up because it looks like something is failing - it's perfectly
normal however :)

Is everything working OK with 1.6-rc?

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@psfales
Copy link

psfales commented Apr 29, 2015

Sigh. This is getting puzzling. I think it worked fine yesterday.
I think I haven't changed anything. But it's not working today.

"weave run" works ok, but when I start a container with WEAVE_CIDR,
the docker run command hangs.

I see this in "docker logs powerstrip"

2015-04-29 19:33:29+0000 [HTTP11ClientProtocol,client] while running chain
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/treq/client.py", line 44, in connectionLost

      File "build/bdist.linux-x86_64/egg/treq/content.py", line 35, in connectionLost

      File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 382, in callback
        self._startRunCallbacks(result)
      File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 490, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/usr/local/lib/python2.7/dist-packages/Twisted-14.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
        raise ValueError("No JSON object could be decoded")
    exceptions.ValueError: No JSON object could be decoded

Peter Fales
Peter "at" fales-lorenz.net

On Wed, Apr 29, 2015 at 04:50:30AM -0700, Kai Davenport wrote:

[1]@psfales The Stopping factory stuff in the logs is the Python
Twisted libraries way of saying "I did a connection" - this always
trips me up because it looks like something is failing - it's perfectly
normal however :)

Is everything working OK with 1.6-rc?

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@binocarlos
Copy link
Owner

@psfales could you post the commands you are using to spin everything up - I will boot a CentOS 7 VM and replicate :-)

@psfales
Copy link

psfales commented Apr 30, 2015

I'm starting to think that something else is going on. There may
be hardware issues or software (network configuration issues). It
seems like I will try tweaking something and then it works, so I put
put it back the way it was, and it still works. Then after a while
it stops working again. It almost seems like it's just random. So,
I'm not sure if the problem is the Weave, Powerstrip, my hardware,
or soemthing else.

But, there is definitely a serious problem with the native CentOS7
version of docker-1.5. It can't do volume mounts of sockets. I've
experimented withe various non-RedHat versions of docker (1.5, 1.6-rc5,
and 1.6-rc6) but haven't been able to get any consistent results (that's
one of the variables where it first works and then doesn't work or
vice versa)

But having said that. Here are command's I'm using...

Launch plugin

unset DOCKER_HOST
docker run -d --name powerstrip-weave
--expose 80
-e http_proxy=http://global.proxy.lucent.com:8000/
-e https_proxy=http://global.proxy.lucent.com:8000/
-e no_proxy=/var/run/docker.sock
-v /var/run/docker.sock:/var/run/docker.sock
docker.io/binocarlos/powerstrip-weave launch $*

Launch Powestrip

docker run -d --name powerstrip
-e no_proxy=/var/run/docker.sock
-v /var/run/docker.sock:/var/run/docker.sock
-v ~/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml
--link powerstrip-weave:weave
-p 2376:2375
docker.io/clusterhq/powerstrip

Launch container

docker -H 127.0.0.1:2376 run -d -e WEAVE_CIDR=10.0.5.1/16 tutum/centos

Sometimes this last command will work. Sometimes it will print
the container id, then hang.

Peter Fales
Peter "at" fales-lorenz.net

On Thu, Apr 30, 2015 at 06:22:24AM -0700, Kai Davenport wrote:

[1]@psfales could you post the commands you are using to spin
everything up - I will boot a CentOS 7 VM and replicate :-)

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@binocarlos
Copy link
Owner

Can confirm that vanilla docker on CentOS 7 does not mount sockets as volumes:

$ sudo docker version
Client version: 1.5.0-dev
Client API version: 1.18
Go version (client): go1.3.3
Git commit (client): fc0329b/1.5.0
OS/Arch (client): linux/amd64
Server version: 1.5.0-dev
Server API version: 1.18
Go version (server): go1.3.3
Git commit (server): fc0329b/1.5.0
OS/Arch (server): linux/amd64
$ sudo docker info
Containers: 1
Images: 51
Storage Driver: devicemapper
 Pool Name: docker-253:1-432553-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.136 GB
 Data Space Total: 107.4 GB
 Data Space Available: 37.73 GB
 Metadata Space Used: 3.502 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 1
Total Memory: 490.4 MiB
Name: localhost.localdomain
ID: 5UMR:SGRM:HLJF:AHJT:7ZBP:AAM7:UC4P:MLIY:Z6AR:DCOX:W45F:VCMV

This means that whilst the powerstrip-weave container itself starts - it is not starting the other containers that are needed.

This can be seen doing docker logs powerstrip-weave

2015/04/30 16:24:04 Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
2015/04/30 16:24:04 Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
server listening on port: 80

The 2 failures to connect to docker is the adapter trying to start wait-for-weave and the weave router itself (using /var/run/docker.sock)

Proceeding from here will guarantee that the WEAVE_CIDR container will hang because weave is not even started.

I will now try another version of docker and see if the same problem persists...

@binocarlos
Copy link
Owner

OK - with a docker 1.6 installed - powerstrip-weave boots but I am now getting the same hung problem.

When I docker logs powerstrip I get:

2015-04-30 17:00:40+0000 [HTTPChannel,0,172.17.42.1] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7f09c6848fc8>
2015-04-30 17:00:40+0000 [HTTP11ClientProtocol,client] while running chain
    Traceback (most recent call last):
    Failure: twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionLost'>>]

Which certainly indicates something going wrong. Good news - I can replicate the problem :-)

Bad news - I'm off home now so won't take a look until tomorrow.

Thanks @psfales for the report though :)

@psfales
Copy link

psfales commented Apr 30, 2015

On Thu, Apr 30, 2015 at 09:28:53AM -0700, Kai Davenport wrote:

Can confirm that vanilla docker on CentOS 7 does not mount sockets as
volumes:

Yes, I have confirmed that.

This means that whilst the powerstrip-weave container itself starts -
it is not starting the other containers that are needed.

This can be seen doing docker logs powerstrip-weave
2015/04/30 16:24:04 Cannot connect to the Docker daemon. Is 'docker -d' running
on this host?
2015/04/30 16:24:04 Cannot connect to the Docker daemon. Is 'docker -d' running

This is the symptom with the vanilla docker - since the container can't
see the socket, it's not able to talk to the daemon.

--
Reply to this email directly or [1]view it on GitHub.

References

  1. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@psfales
Copy link

psfales commented Apr 30, 2015

Thansk for letting me know the status. And I'm glad to hear that
you are able to reproduce the problem. For what it's worth, I've seen
it work sometimes without hanging - So far I haven't figured out the reason
why it sometimes works and sometimes does not.

On Thu, Apr 30, 2015 at 10:03:05AM -0700, Kai Davenport wrote:

OK - with a docker 1.6 installed - powerstrip-weave boots but I am now
getting the same hung problem.

When I docker logs powerstrip I get:
2015-04-30 17:00:40+0000 [HTTPChannel,0,172.17.42.1] Starting factory <twisted.w
eb.client._HTTP11ClientFactory instance at 0x7f09c6848fc8>
2015-04-30 17:00:40+0000 [HTTP11ClientProtocol,client] while running chain
Traceback (most recent call last):
Failure: twisted.web._newclient.ResponseNeverReceived: [<twisted.python.fail
ure.Failure <class 'twisted.internet.error.ConnectionLost'>>]

Which certainly indicates something going wrong. Good news - I can
replicate the problem :-)

Bad news - I'm off home now so won't take a look until tomorrow.

Thanks [1]@psfales for the report though :)

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@binocarlos
Copy link
Owner

Hey - so yes I see what you mean - this morning, having spun up everything, it all works fine now :-/

Kernel: 3.10.0-123.13.1.el7.x86_64 - OS: CentOS Linux 7 (Core)

I have this script to run through all the steps:

#!/usr/bin/env bash
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi
setenforce 0
yum update -y
yum install -y docker
wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O /usr/bin/docker
service docker start
docker pull docker.io/binocarlos/powerstrip-weave
docker pull docker.io/clusterhq/powerstrip
docker pull tutum/centos
mkdir -p /tmp/powerstrip-demo
cat > /tmp/powerstrip-demo/adapters.yml <<EOF
endpoints:
  "POST /*/containers/create":
    pre: [weave]
  "POST /*/containers/*/start":
    post: [weave]
  "POST /*/containers/*/restart":
    post: [weave]
adapters:
  weave: http://weave/v1/extension
EOF
unset DOCKER_HOST
# to debug - run this container in a seperate shell
# it mounts the source code
# first command: bash /srv/app/run.sh launch
# ctrl + c
# bash /srv/app/run.sh softlaunch
# ^^ repeat
docker run -d --name powerstrip-weave \
   --expose 80 \
   -e http_proxy=http://global.proxy.lucent.com:8000/ \
   -e https_proxy=http://global.proxy.lucent.com:8000/ \
   -e no_proxy=/var/run/docker.sock \
   -v /srv/projects/powerstrip-weave:/srv/app \
   -v /var/run/docker.sock:/var/run/docker.sock \
    docker.io/binocarlos/powerstrip-weave launch
docker run -d --name powerstrip \
  -e no_proxy=/var/run/docker.sock \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml \
  --link powerstrip-weave:weave \
  -p 2376:2375 \
  docker.io/clusterhq/powerstrip
docker  -H 127.0.0.1:2376 run -d -e WEAVE_CIDR=10.0.5.1/16 tutum/centos
docker exec -ti weavetest ip addr

Having run this 3 times - each time it outputs:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
21: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:07 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.7/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:7/64 scope link
       valid_lft forever preferred_lft forever
23: ethwe: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc pfifo_fast state UP qlen 1000
    link/ether 2e:f8:c0:3a:8b:0b brd ff:ff:ff:ff:ff:ff
    inet 10.0.5.1/16 scope global ethwe
       valid_lft forever preferred_lft forever
    inet6 fe80::2cf8:c0ff:fe3a:8b0b/64 scope link
       valid_lft forever preferred_lft forever

Can you try the same script from a fresh CentOS7 box?

@psfales Thanks for spending time on tracking this down :)

@psfales
Copy link

psfales commented May 1, 2015

Do you mean that I can simply put that script into a file and run it?

I had to make some tweaks to get it workign behind our corporate
firewall, but there are a few other issues too...

  • What is the "-v /srv/projects/powerstrip-weave:/srv/app" directory? I
    don't have that.
  • Do you need some delay between starting docker, starting the plugin, and
    and starting powerstrip?
  • What is "weavetest" in the last line? Is that the tutum/centos
    container?

Peter Fales
Peter "at" fales-lorenz.net

On Fri, May 01, 2015 at 04:01:03AM -0700, Kai Davenport wrote:

Hey - so yes I see what you mean - this morning, having spun up
everything, it all works fine now :-/

Kernel: 3.10.0-123.13.1.el7.x86_64 - OS: CentOS Linux 7 (Core)

I have this script to run through all the steps:

#!/usr/bin/env bash
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
setenforce 0
yum update -y
yum install -y docker
wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O /usr/bin/docker
service docker start
docker pull docker.io/binocarlos/powerstrip-weave
docker pull docker.io/clusterhq/powerstrip
docker pull tutum/centos
mkdir -p /tmp/powerstrip-demo
cat > /tmp/powerstrip-demo/adapters.yml <<EOF
endpoints:
"POST //containers/create":
pre: [weave]
"POST /
/containers//start":
post: [weave]
"POST /
/containers/*/restart":
post: [weave]
adapters:
weave: http://weave/v1/extension
EOF
unset DOCKER_HOST

to debug - run this container in a seperate shell

it mounts the source code

first command: bash /srv/app/run.sh launch

ctrl + c

bash /srv/app/run.sh softlaunch

^^ repeat

docker run -d --name powerstrip-weave
--expose 80
-e http_proxy=http://global.proxy.lucent.com:8000/
-e https_proxy=http://global.proxy.lucent.com:8000/
-e no_proxy=/var/run/docker.sock
-v /srv/projects/powerstrip-weave:/srv/app
-v /var/run/docker.sock:/var/run/docker.sock
docker.io/binocarlos/powerstrip-weave launch
docker run -d --name powerstrip
-e no_proxy=/var/run/docker.sock
-v /var/run/docker.sock:/var/run/docker.sock
-v /tmp/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml
--link powerstrip-weave:weave
-p 2376:2375
docker.io/clusterhq/powerstrip
docker -H 127.0.0.1:2376 run -d -e WEAVE_CIDR=10.0.5.1/16 tutum/centos
docker exec -ti weavetest ip addr

Having run this 3 times - each time it outputs:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
21: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:07 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.7/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe11:7/64 scope link
valid_lft forever preferred_lft forever
23: ethwe: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc pfifo_fast state UP
qlen 1000
link/ether 2e:f8:c0:3a:8b:0b brd ff:ff:ff:ff:ff:ff
inet 10.0.5.1/16 scope global ethwe
valid_lft forever preferred_lft forever
inet6 fe80::2cf8:c0ff:fe3a:8b0b/64 scope link
valid_lft forever preferred_lft forever

Can you try the same script from a fresh CentOS7 box?

[1]@psfales Thanks for spending time on tracking this down :)

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@psfales
Copy link

psfales commented May 1, 2015

My slightly modified script is attached. I tried it on four different
CentOS 7 machines. On three of them it ran 10 times in a row without
errors. On the fourth one, it was consistently failing. The I rebooted
that failing machine and it started working again (ran 10 times without
errors)

Prior to the reboot, I realized I was running several qemu-kvm VMs. Maybe
those where doing something with the network stack, maybe they were
just slowing things down a bit, or maybe it's something else entireley

I noticed that it pulled newer versions of powerstrip and
powerstrip-weave the first time it ran. So I guess the versions
I had been been using previously were somewhat out of date.

I also noticed some evidence that timing is a factor. The last
30 second sleep is important, although the failure signature is
different when I leave it out.

Peter Fales
Peter "at" fales-lorenz.net

On Fri, May 01, 2015 at 04:01:03AM -0700, Kai Davenport wrote:

Hey - so yes I see what you mean - this morning, having spun up
everything, it all works fine now :-/

Kernel: 3.10.0-123.13.1.el7.x86_64 - OS: CentOS Linux 7 (Core)

I have this script to run through all the steps:

#!/usr/bin/env bash
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
setenforce 0
yum update -y
yum install -y docker
wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O /usr/bin/docker
service docker start
docker pull docker.io/binocarlos/powerstrip-weave
docker pull docker.io/clusterhq/powerstrip
docker pull tutum/centos
mkdir -p /tmp/powerstrip-demo
cat > /tmp/powerstrip-demo/adapters.yml <<EOF
endpoints:
"POST //containers/create":
pre: [weave]
"POST /
/containers//start":
post: [weave]
"POST /
/containers/*/restart":
post: [weave]
adapters:
weave: http://weave/v1/extension
EOF
unset DOCKER_HOST

to debug - run this container in a seperate shell

it mounts the source code

first command: bash /srv/app/run.sh launch

ctrl + c

bash /srv/app/run.sh softlaunch

^^ repeat

docker run -d --name powerstrip-weave
--expose 80
-e http_proxy=http://global.proxy.lucent.com:8000/
-e https_proxy=http://global.proxy.lucent.com:8000/
-e no_proxy=/var/run/docker.sock
-v /srv/projects/powerstrip-weave:/srv/app
-v /var/run/docker.sock:/var/run/docker.sock
docker.io/binocarlos/powerstrip-weave launch
docker run -d --name powerstrip
-e no_proxy=/var/run/docker.sock
-v /var/run/docker.sock:/var/run/docker.sock
-v /tmp/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml
--link powerstrip-weave:weave
-p 2376:2375
docker.io/clusterhq/powerstrip
docker -H 127.0.0.1:2376 run -d -e WEAVE_CIDR=10.0.5.1/16 tutum/centos
docker exec -ti weavetest ip addr

Having run this 3 times - each time it outputs:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
21: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:07 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.7/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe11:7/64 scope link
valid_lft forever preferred_lft forever
23: ethwe: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc pfifo_fast state UP
qlen 1000
link/ether 2e:f8:c0:3a:8b:0b brd ff:ff:ff:ff:ff:ff
inet 10.0.5.1/16 scope global ethwe
valid_lft forever preferred_lft forever
inet6 fe80::2cf8:c0ff:fe3a:8b0b/64 scope link
valid_lft forever preferred_lft forever

Can you try the same script from a fresh CentOS7 box?

[1]@psfales Thanks for spending time on tracking this down :)

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@binocarlos
Copy link
Owner

@psfales Hey - sorry I had pasted the script with some artifacts left from me playing around.- more stupidly I hadn't ran it as a script instead I built up the commands as I ran them manually.

  • the -v /srv/projects/powerstrip-weave:/srv/app can go - it was to allow me to put logging in the adapter without git commit / docker build etc
  • a sleep 5 would be a good idea (because I was running it manually)
  • weavetest is indeed the tutum/centos container - a --name weavetest is needed on the previous line

Can you paste your modified script? I can't find the github email attachment.

I'd be interested in seeing if it works on the rebooted machine consistently? Thanks again!

@psfales
Copy link

psfales commented May 1, 2015

The complete script is pasted below. Yes, it seems to run consistently
on the rebooted machine. (I ran it 10 times in a row along with 10
times on the three other machines.)

The question is whether it will still be working on Monday!

#!/usr/bin/env bash
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
setenforce 0
yum update -y
yum install -y docker
service docker stop
export https_proxy=http://global.proxy.lucent.com:8000/
wget -c --no-check-certificate https://get.docker.com/builds/Linux/x86_64/docker-latest -O /usr/bin/docker
service docker start
docker rm -f $(docker ps -qa)
docker pull docker.io/binocarlos/powerstrip-weave
docker pull docker.io/clusterhq/powerstrip
docker pull tutum/centos
mkdir -p /tmp/powerstrip-demo
cat > /tmp/powerstrip-demo/adapters.yml <<EOF
endpoints:
"POST //containers/create":
pre: [weave]
"POST /
/containers//start":
post: [weave]
"POST /
/containers/*/restart":
post: [weave]
adapters:
weave: http://weave/v1/extension
EOF
unset DOCKER_HOST

to debug - run this container in a seperate shell

it mounts the source code

first command: bash /srv/app/run.sh launch

ctrl + c

bash /srv/app/run.sh softlaunch

^^ repeat

-v /srv/projects/powerstrip-weave:/srv/app \

docker run -d --name powerstrip-weave
--expose 80
-e http_proxy=http://global.proxy.lucent.com:8000/
-e https_proxy=http://global.proxy.lucent.com:8000/
-e no_proxy=/var/run/docker.sock
-v /var/run/docker.sock:/var/run/docker.sock
docker.io/binocarlos/powerstrip-weave launch
docker run -d --name powerstrip
-e no_proxy=/var/run/docker.sock
-v /var/run/docker.sock:/var/run/docker.sock
-v /tmp/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml
--link powerstrip-weave:weave
-p 2376:2375
docker.io/clusterhq/powerstrip
sleep 30
N=${1:-1}
docker -H 127.0.0.1:2376 run -d --name weavetest -e WEAVE_CIDR=10.0.5.${N}/16 tutum/centos
sleep 1

docker exec -ti weavetest ip addr

Peter Fales
Peter "at" fales-lorenz.net

On Fri, May 01, 2015 at 12:52:55PM -0700, Kai Davenport wrote:

[1]@psfales Hey - sorry I had pasted the script with some artifacts
left from me playing around.- more stupidly I hadn't ran it as a script
instead I built up the commands as I ran them manually.
* the -v /srv/projects/powerstrip-weave:/srv/app can go - it was to
allow me to put logging in the adapter without git commit / docker
build etc
* a sleep 5 would be a good idea (because I was running it manually)
* weavetest is indeed the tutum/centos container - a --name weavetest
is needed on the previous line

Can you paste your modified script? I can't find the github email
attachment.

I'd be interested in seeing if it works on the rebooted machine
consistently? Thanks again!

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@errordeveloper
Copy link
Collaborator Author

@psfales this script disabled SELinux, have you, by chance, had it enabled when you observed inconsistent behaviour?

@psfales
Copy link

psfales commented May 3, 2015

That was in the script that Kai provided, but I don't think it had
any effect in this case. We have selinux turned off on all of our
systems.

Peter Fales
Peter "at" fales-lorenz.net

On Fri, May 01, 2015 at 04:33:33PM -0700, Ilya Dmitrichenko wrote:

[1]@psfales this script disabled SELinux, have you, by chance, had it
enabled when you observed inconsistent behaviour?

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@psfales
Copy link

psfales commented May 4, 2015

For the record, I ran through the script at least 10 times on each
of 4 different machines, and it run perfectly every time. (I don't
know if that's good new or bad news!)

Peter Fales
Peter "at" fales-lorenz.net

On Fri, May 01, 2015 at 04:01:03AM -0700, Kai Davenport wrote:

Hey - so yes I see what you mean - this morning, having spun up
everything, it all works fine now :-/

Kernel: 3.10.0-123.13.1.el7.x86_64 - OS: CentOS Linux 7 (Core)

I have this script to run through all the steps:

#!/usr/bin/env bash
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
setenforce 0
yum update -y
yum install -y docker
wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O /usr/bin/docker
service docker start
docker pull docker.io/binocarlos/powerstrip-weave
docker pull docker.io/clusterhq/powerstrip
docker pull tutum/centos
mkdir -p /tmp/powerstrip-demo
cat > /tmp/powerstrip-demo/adapters.yml <<EOF
endpoints:
"POST //containers/create":
pre: [weave]
"POST /
/containers//start":
post: [weave]
"POST /
/containers/*/restart":
post: [weave]
adapters:
weave: http://weave/v1/extension
EOF
unset DOCKER_HOST

to debug - run this container in a seperate shell

it mounts the source code

first command: bash /srv/app/run.sh launch

ctrl + c

bash /srv/app/run.sh softlaunch

^^ repeat

docker run -d --name powerstrip-weave
--expose 80
-e http_proxy=http://global.proxy.lucent.com:8000/
-e https_proxy=http://global.proxy.lucent.com:8000/
-e no_proxy=/var/run/docker.sock
-v /srv/projects/powerstrip-weave:/srv/app
-v /var/run/docker.sock:/var/run/docker.sock
docker.io/binocarlos/powerstrip-weave launch
docker run -d --name powerstrip
-e no_proxy=/var/run/docker.sock
-v /var/run/docker.sock:/var/run/docker.sock
-v /tmp/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml
--link powerstrip-weave:weave
-p 2376:2375
docker.io/clusterhq/powerstrip
docker -H 127.0.0.1:2376 run -d -e WEAVE_CIDR=10.0.5.1/16 tutum/centos
docker exec -ti weavetest ip addr

Having run this 3 times - each time it outputs:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
21: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:07 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.7/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe11:7/64 scope link
valid_lft forever preferred_lft forever
23: ethwe: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc pfifo_fast state UP
qlen 1000
link/ether 2e:f8:c0:3a:8b:0b brd ff:ff:ff:ff:ff:ff
inet 10.0.5.1/16 scope global ethwe
valid_lft forever preferred_lft forever
inet6 fe80::2cf8:c0ff:fe3a:8b0b/64 scope link
valid_lft forever preferred_lft forever

Can you try the same script from a fresh CentOS7 box?

[1]@psfales Thanks for spending time on tracking this down :)

--
Reply to this email directly or [2]view it on GitHub.

References

  1. https://github.com/psfales
  2. /usr/bin/docker requires libdevmapper.so on CoreOS and other versions of Linux #8 (comment)

@cdancy
Copy link

cdancy commented May 14, 2015

Coreos user here. Had the same issue and just mounted libdevmapper in as noted in the below article and it worked as expected.

https://groups.google.com/forum/#!topic/coreos-user/4nyJh8lYaiA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants