Macosx default machine issues with nested podman.

[james-lawrence]

Issue Description

Attempting to run nested podman containers in the macosx podman-default-machine and running into permissions issues when writing the containers to a mounted folder:

podman run --name derp-test --detach --replace --userns host --privileged --volume /Users/derp/.cache/eg/containers:/var/lib/containers:rw quay.io/podman/machine-os:5.3 /usr/sbin/init
podman exec -it derp-test /bin/bash

podman run -it ubuntu:oracular /bin/bash
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:oracular...
Getting image source signatures
Copying blob 6d229850a778 done   | 
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:6d229850a7784435237486a1406ffb5a4f0a4ac3b993928d18b30ad6853c6845"/""/"sha256:ff8ef9b034b737e0182308bb341f5f751c39b59779a60356672afe015e8423a3": unpacking failed (error: exit status 1; output: setting up pivot dir: mkdir ./.pivot_root1056252533: permission denied)

### running these two commands manually afterwards work fine.
mkdir -p /var/lib/containers/storage/overlay/ff8ef9b034b737e0182308bb341f5f751c39b59779a60356672afe015e8423a3/diff
echo "fooo!" > /var/lib/containers/storage/overlay/ff8ef9b034b737e0182308bb341f5f751c39b59779a60356672afe015e8423a3/diff/foo.txt
cat /var/lib/containers/storage/overlay/ff8ef9b034b737e0182308bb341f5f751c39b59779a60356672afe015e8423a3/diff/foo.txt
fooo!

Steps to reproduce the issue

1. podman machine init --now
2. podman run --name derp-test --detach --replace --userns host --privileged --volume /Users/derp/.cache/eg/containers:/var/lib/containers:rw quay.io/podman/machine-os:5.3 /usr/sbin/init
3. podman exec -it derp-test /bin/bash
4. podman run -it ubuntu:oracular /bin/bash

Describe the results you received

doesnt work; but works fine on linux systems:

podman run --name derp-test --detach --replace --userns host --privileged --volume ~/.cache/eg/containers:/var/lib/containers:rw quay.io/podman/machine-os:5.3 /usr/sbin/init
podman exec -it derp-test /bin/bash
podman run -it ubuntu:oracular /bin/bash
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:oracular...
Getting image source signatures
Copying blob 31734b193a81 done   | 
Copying config e40b6e31bd done   | 
Writing manifest to image destination

Describe the results you expected

expected the container to be pulled and stored successfully.

podman info output

host:
podman info
host:
  arch: arm64
  buildahVersion: 1.38.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 91.75
    systemPercent: 3.74
    userPercent: 4.51
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "41"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.12.7-200.fc41.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 602607616
  memTotal: 2042306560
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1.fc41.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.1-1.fc41.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.1
  ociRuntime:
    name: crun
    package: crun-1.19.1-1.fc41.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.19.1
      commit: 3e32a70c93f5aa5fea69b50256cca7fd4aa23c80
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20241211.g09478d5-1.fc41.aarch64
    version: |
      pasta 0^20241211.g09478d5-1.fc41.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: unix:///run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.fc41.aarch64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 4m 3.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106415992832
  graphRootUsed: 5763751936
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 20
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.2
  Built: 1737504000
  BuiltTime: Tue Jan 21 18:00:00 2025
  GitCommit: ""
  GoVersion: go1.23.4
  Os: linux
  OsArch: linux/arm64
  Version: 5.3.2


inside the vm:

host:
  arch: arm64
  buildahVersion: 1.38.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 93.33
    systemPercent: 3.28
    userPercent: 3.39
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: 651b24746668
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.12.7-200.fc41.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 237178880
  memTotal: 2042306560
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.aarch64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 3m 40.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 126562507685888
  graphRootUsed: 79088314220544
  graphStatus:
    Backing Filesystem: <unknown>
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.3.1
  Built: 1732147200
  BuiltTime: Thu Nov 21 00:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/arm64
  Version: 5.3.1

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

macosx host. applehv provider.

Additional information

pretty much at a loss for this atm. these fixes might resolve it? its unclear.

in the podman default machine it looks like the Options typo was fixed. 🤷

[Luap99]

--volume /Users/derp/.cache/eg/containers:/var/lib/containers:rw

You cannot use virtiofs as container storage path which means you cannot use host paths for them. I suggest you use a normal volume instead

[james-lawrence]

I'm sorry, not following. this works fine on linux hosts. what is the technical issue here that prevents it from working on macosx? aka what is podman/crun/whatever doing differently that results in the difference in behavior I manually create files but get permission denied when podman tries to write to the directory?

What is your definition of a 'normal' volume vs --volume src:dst:rw?

Basically, if i go strace what am I going to find podman doing that causes it to fail here.

[Luap99]

Because you are not using virtiofs on your normal linux. Like I said the host paths that ar emounted on th emachine VM are not suitable for a contianer store. If you mount another path from the VM it can work. The easiest is to use podman volumes
podman volume create testvol then -v testvol:/var/lib/containers ... Or just leave the volume create command as the volume will be created automatically by run if it does not exists.

[james-lawrence]

What are the limitations with virtiofs that cause this? again these are straight up writes to the directory and I can write to these folders just fine from the same environment were podman build fails, its the podman command having the issue. It doesn't seem to me that virtiofs is problematic, it seems like podman and the underlying code/libraries are doing something odd.

As a end user (and developer) this looks 100% like a bug both the mismatch in behavior between host OS's and the mismatch in behavior between podman and other programs on the macosx host.

edit: please note: I appreciate the suggestions for alternative approaches. but they require more complexity for end users or more specialization by developers. I'm trying to drill into why podman is behaving this way and determining if its an actual technical limitation or just 'its hard'. From a personal point of view I don't see why there would be a difference between creating a volume via podman which at its core is still FS managed by the host and just using --volume directly on a directory.