Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: RYUK container startup failure when running Docker in rootless mode #537

Open
keneanung opened this issue Apr 12, 2024 · 8 comments · May be fixed by #710
Open

Bug: RYUK container startup failure when running Docker in rootless mode #537

keneanung opened this issue Apr 12, 2024 · 8 comments · May be fixed by #710

Comments

@keneanung
Copy link

keneanung commented Apr 12, 2024 •
edited by alexanderankin
Loading

Describe the bug

When Docker is run in rootless mode, the ryuk fails to start as the docker socket mounted as a volume has the wrong permissions (nobody:nobody). This can be fixed be mounting the correct socket at /run/user/$(id -u)/docker.sock.

The overwrite can be done by setting the environment variable TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE to the above mentioned socket.

Ideally, this would be detected and configured automatically (ie. using Server.Security Options.rootless and Endpoints.docker.Host of the active context or the DOCKER_HOST environment variable).

If this is not an option, please consider documenting this case more in detail.

To Reproduce

Configure Docker to run in rootless mode (see https://docs.docker.com/engine/security/rootless/) and run the following snippet:

>>> from testcontainers.postgres import PostgresContainer
>>> with PostgresContainer() as container:
...   assert True

Runtime environment

Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.

# Get the operating system information (on a unix os).
$ uname -a
Linux FE-C-012RG 6.5.0-1019-oem #20-Ubuntu SMP PREEMPT_DYNAMIC Mon Mar 18 17:38:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
# Get the python version.
$ python --version
Python 3.12.2
# Get the docker version and other docker information.
$ docker info
Client: Docker Engine - Community
 Version:    26.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 19
 Server Version: 26.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.5.0-1019-oem
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.32GiB
 Name: FE-C-012RG
 ID: 547c075a-39fa-4b4a-950a-b18069861839
 Docker Root Dir: /home/CFL5FE/.local/share/docker
 Debug Mode: false
 Username: keneanung
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
# Get all python packages.
$ pip freeze
alembic==1.13.1
annotated-types==0.6.0
anyio==4.3.0
astroid==3.1.0
asyncpg==0.29.0
-e <my local repo>
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
coverage==7.4.4
cryptography==42.0.5
dill==0.3.8
docker==7.0.0
ecdsa==0.19.0
fastapi==0.110.1
fastapi-azure-auth==4.3.1
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
hvac==2.1.0
idna==3.6
importlib_metadata==7.1.0
iniconfig==2.0.0
isort==5.13.2
Mako==1.3.2
MarkupSafe==2.1.5
mccabe==0.7.0
packaging==24.0
platformdirs==4.2.0
pluggy==1.4.0
pyasn1==0.6.0
pycparser==2.22
pydantic==2.6.4
pydantic_core==2.16.3
pylint==3.1.0
pytest==8.1.1
pytest-asyncio==0.23.6
pytest-cov==5.0.0
python-jose==3.3.0
PyYAML==6.0.1
requests==2.31.0
rsa==4.9
six==1.16.0
sniffio==1.3.1
SQLAlchemy==2.0.29
starlette==0.37.2
syrupy==4.6.1
testcontainers==4.3.3
tomli==2.0.1
tomlkit==0.12.4
typing_extensions==4.11.0
urllib3==2.2.1
uvicorn==0.29.0
vault-env-gen==0.2.0
wrapt==1.16.0
yapf==0.40.2
zipp==3.18.1
@alexanderankin
Copy link
Member

can you confirm this approach:

mkdir test-docker-rootless-detection ; cd $_ ; python -m venv .venv && . $_/bin/activate
pip install docker
cat >  detect_rootless.py <<EOF
from docker import from_env
from docker.client import DockerClient
from docker.models.containers import Container, ContainerCollection


def is_rootless(client: DockerClient):
  info = client.info()
  sec_opts = info.get('SecurityOptions') or tuple()

  return any('rootless' in s for s in sec_opts)


if __name__ == "__main__":
  print(is_rootless(from_env()))
EOF

python detect_rootless.py

@alexanderankin
Copy link
Member

and then i guess we will need to tweak the Reaper class a bit in core

@keneanung
Copy link
Author

can you confirm this approach:

mkdir test-docker-rootless-detection ; cd $_ ; python -m venv .venv && . $_/bin/activate
pip install docker
cat >  detect_rootless.py <<EOF
from docker import from_env
from docker.client import DockerClient
from docker.models.containers import Container, ContainerCollection


def is_rootless(client: DockerClient):
  info = client.info()
  sec_opts = info.get('SecurityOptions') or tuple()

  return any('rootless' in s for s in sec_opts)


if __name__ == "__main__":
  print(is_rootless(from_env()))
EOF

python detect_rootless.py

This returns the following:

$ python detect_rootless.py 
True

@flonix8
Copy link

flonix8 commented Aug 20, 2024

I also use rootless Docker and I can confirm this issue.
I can also confirm that setting TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/$(id -u)/docker.sock fixes the issue. Alternatively, I have tried disabling Ryuk through TESTCONTAINERS_RYUK_DISABLED=true, which makes the tests run flawlessly as well.

@vemonet
Copy link

vemonet commented Oct 1, 2024 •
edited
Loading

Would it be possible to fix this so that testcontainers just works with rootless docker? @alexanderankin

TESTCONTAINERS_RYUK_DISABLED=true pytest works, but it's a pain to use, and we can't even just os.environ['TESTCONTAINERS_RYUK_DISABLED'] = 'true' inside the python test file

Nowadays any dev slightly conscious should be using rootless docker (because it is safer, and because it reduce friction with file ownerships when mounting volumes inside an image that use the root user. So not only it is safer, but it is also easier to use...). It's a bit the future of containers, but it's already there and working. So testcontainers should be available for rootless docker without the need for complex setup

@alexanderankin
Copy link
Member

you can import the config module and set it on the dataclass there

testcontainers-python/core/testcontainers/core/config.py

Line 51 in 85d6078

ryuk_disabled: bool = RYUK_DISABLED

  • I'm not in a position where I'm forced to use rootless docker so I dont have easy access to an environment where I could easily test. if there is a guide or something that I can set up, this would be good prereq for contributions imo

@alexanderankin
Copy link
Member

Ah, we have confirmed that the logic i provided above works. Then all that remains is plugging it in I suppose. I'll accept PR that fixes the issue. yes.

@vemonet
Copy link

vemonet commented Oct 2, 2024

I am not forced to use docker rootless neither, it's just that it's better than the root option, especially when you use containers a lot for development, try it you'll see :) Rootless docker can be easily enabled following these docs: https://docs.docker.com/engine/security/rootless/ usually 3 commands:

dockerd-rootless-setuptool.sh install
systemctl --user enable docker
loginctl enable-linger $UID

Unfortunately I just realized that ryuk is needed to stop the containers after running the tests. So disabling it is not a sustainable solution

But TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/$(id -u)/docker.sock works fine

An easy solution would be to just change the default RYUK_DOCKER_SOCKET depending on if rootless is detected or not here: https://github.com/testcontainers/testcontainers-python/blob/main/core/testcontainers/core/config.py#L15

I might look into this if I find sometimes and will send a PR

vemonet added a commit to vemonet/testcontainers-python that referenced this issue Oct 2, 2024
@vemonet
fix(core): Fix retrieval of the docker socket path when using rootles…
bba928f 
…s docker (required to run ryuk). Fixes testcontainers#537
@vemonet vemonet linked a pull request Oct 2, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@keneanung @vemonet @alexanderankin @flonix8