Play fails if temp directory is deleted mid-play #1061

markafarrell · 2024-04-12T06:04:21Z

If the ansible temp directory is removed mid-play mitogen does not recreate it and the play fails.

An exception occurred during task execution. To see the full traceback, use -vvv. The error was:     _os.mkdir(file, 0o700)
fatal: [172.17.0.9]: FAILED! => {"msg": "Unexpected failure during module execution: builtins.FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.ansible-test/tmp/ansible_mitogen_runner_y_absj50'
  File \"<stdin>\", line 3876, in _dispatch_one
  File \"master:/home/xxxxxxxx/work/mitogen-repro/.venv/lib/python3.10/site-packages/ansible_mitogen/target.py\", line 415, in run_module
    return impl.run()
           ^^^^^^^^^^
  File \"master:/home/d384492/work/mitogen-repro/.venv/lib/python3.10/site-packages/ansible_mitogen/runner.py\", line 445, in run
    self.setup()
  File \"master:/home/d384492/work/mitogen-repro/.venv/lib/python3.10/site-packages/ansible_mitogen/runner.py\", line 934, in setup
    self._stdio = NewStyleStdio(self.args, self.get_temp_dir())
                                           ^^^^^^^^^^^^^^^^^^^
  File \"master:/home/d384492/work/mitogen-repro/.venv/lib/python3.10/site-packages/ansible_mitogen/runner.py\", line 361, in get_temp_dir
    self._temp_dir = tempfile.mkdtemp(
                     ^^^^^^^^^^^^^^^^^
  File \"/usr/lib/python3.11/tempfile.py\", line 507, in mkdtemp
    _os.mkdir(file, 0o700)
", "stdout": ""}

Using the normal ansible strategy the temp directory is recreated and the play succeeds.

Ansible version: 2.14.15

Host OS: Ubuntu (WSL2)
Target OS: Debian12 (docker)

Host Python: Python 3.10.12
Target Python: Python 3.11.2

See https://github.com/markafarrell/mitogen-repro-issue-1061 for reproduction instructions

The text was updated successfully, but these errors were encountered:

moreati · 2024-04-21T09:46:53Z

I'm attempting to reproduce this. Step 4 of https://github.com/markafarrell/mitogen-repro-issue-1061 doesn't leave a running container. Instead it immediately exits.

alex@ubuntu2004:~/mitogen-repro-issue-1061$ docker run -dt --name target-server \
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    --privileged \
    --rm \
    geerlingguy/docker-debian12-ansible:latest;

964532f2b017d53a6292b476e5e463e5157f8520db7e0a6ca6e4d3d3176885ee
alex@ubuntu2004:~/mitogen-repro-issue-1061$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
alex@ubuntu2004:~/mitogen-repro-issue-1061$ docker --version
Docker version 24.0.5, build 24.0.5-0ubuntu1~22.04.1
alex@ubuntu2004:~/mitogen-repro-issue-1061$ uname -a
Linux ubuntu2004 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

markafarrell · 2024-04-22T04:13:53Z

I'm guessing that you are using aarch64 is probably the issue.

There is an arm64 version of that image so it should work.

Do you get anything from:

docker logs target-server

moreati · 2024-04-22T08:37:30Z

alex@ubuntu2004:~/mitogen-repro-issue-1061$ docker rm target-server 
target-server
alex@ubuntu2004:~/mitogen-repro-issue-1061$ docker run -dt --name target-server -v /sys/fs/cgroup:/sys/fs/cgroup:ro --privileged geerlingguy/docker-debian12-ansible:latest;
dea854a953ce1386fcf0ca7b5a28065b5749c982dab711e98fb7210f5968ba39

alex@ubuntu2004:~/mitogen-repro-issue-1061$ docker logs target-server
systemd 252.22-1~deb12u1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization docker.
Detected architecture arm64.

Welcome to Debian GNU/Linux 12 (bookworm)!

Failed to create /init.scope control group: Read-only file system
Failed to allocate manager object: Read-only file system
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...

markafarrell · 2024-04-22T08:50:27Z

Can you try adding --cgroupns=host and change the mount to be rw?

https://serverfault.com/questions/1053187/systemd-fails-to-run-in-a-docker-container-when-using-cgroupv2-cgroupns-priva

moreati · 2024-04-22T09:23:43Z

That did it, and I see the _os.mkdir(file, 0o700) error. Which leads to the next questions

Why don't the unit and integration tests see this? Which extra ingredient(s) matter - Debian 12? systemd? Something Jeff Geerling added?
Can we reproduce it with the existing Mitogen CI images and/or the localhost test?

markafarrell · 2024-04-29T02:57:27Z

Why don't the unit and integration tests see this? Which extra ingredient(s) matter - Debian 12? systemd? Something Jeff Geerling added?

So I think this will happen regardless of OS, systemd etc. The issue is that https://github.com/mitogen-hq/mitogen/blob/master/ansible_mitogen/runner.py#L361 we are essentially doing

mkdir {{ ansible_remote_tmp }}/ansible_mitogen_runner_{{ random stuff }}/

If ansible_remote_tmp doesn't exist this fails.

The existence of this (ansible_remote_tmp) is only checked once, just after we connect to the target, so if it is removed after the connection happens then we see this failure.

2. Can we reproduce it with the existing Mitogen CI images and/or the localhost test?

It should be very easy to reproduce for both localhost and any other image by using a playbook similar to what i have in my reproduction repo. If you can point me to where the test should live i can quickly create one.

moreati · 2024-04-29T10:20:28Z

There are unit tests that mention is_good_temp() in

mitogen/tests/ansible/tests/target_test.py

Line 31 in bb9c51b

class FindGoodTempDirTest(testlib.TestCase):

.
Integration tests should probably be added amongst https://github.com/mitogen-hq/mitogen/blob/bb9c51b3e9cc39fceddd55578bb89680fa4e1acc/tests/ansible/integration/runner/all.yml.

For running tests I'm relying on the Azure CI, and (force) pushing changes. We can squash any interim/WIP commits afterwards.

moreati · 2024-05-19T09:55:18Z

Why don't the unit and integration tests see this? Which extra ingredient(s) matter - Debian 12? systemd? Something Jeff Geerling added?

A factor I previously missed: the repro playbook in https://github.com/markafarrell/mitogen-repro-issue-1061/blob/262591aecadb3ae255c904de17617519f8389673/playbook.yml is explicitly deleting $ANSIBLE_REMOTE_TMP, it's not systemd or similar doing it behind the scenes. There's much less mystery here than I thought, if any.

markafarrell added affects-0.3 Issues related to 0.3.X Mitogen releases bug Code feature that hinders desired execution outcome labels Apr 12, 2024

markafarrell linked a pull request Apr 12, 2024 that will close this issue

Recreate remote temp dir if it no longer exists #1062

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Play fails if temp directory is deleted mid-play #1061

Play fails if temp directory is deleted mid-play #1061

markafarrell commented Apr 12, 2024 •

edited

Loading

moreati commented Apr 21, 2024 •

edited

Loading

markafarrell commented Apr 22, 2024

moreati commented Apr 22, 2024 •

edited

Loading

markafarrell commented Apr 22, 2024

moreati commented Apr 22, 2024

markafarrell commented Apr 29, 2024 •

edited

Loading

moreati commented Apr 29, 2024

moreati commented May 19, 2024

Play fails if temp directory is deleted mid-play #1061

Play fails if temp directory is deleted mid-play #1061

Comments

markafarrell commented Apr 12, 2024 • edited Loading

moreati commented Apr 21, 2024 • edited Loading

markafarrell commented Apr 22, 2024

moreati commented Apr 22, 2024 • edited Loading

markafarrell commented Apr 22, 2024

moreati commented Apr 22, 2024

markafarrell commented Apr 29, 2024 • edited Loading

moreati commented Apr 29, 2024

moreati commented May 19, 2024

markafarrell commented Apr 12, 2024 •

edited

Loading

moreati commented Apr 21, 2024 •

edited

Loading

moreati commented Apr 22, 2024 •

edited

Loading

markafarrell commented Apr 29, 2024 •

edited

Loading