Unable to use custom component after upgrade to 2024.12.0

[antst]

The problem

Custom component which depends on ['questdb>=1.2,<2.0'] can not be used, because installation of questdb package fails with strange error. Here I provide log from manual attempt to install it, automatic installation during startup of core hits the same issue. With manual I just can produce more verbose log.

homeassistant:/config# pip install -v questdb==2.0
Using pip 24.3.1 from /usr/local/lib/python3.13/site-packages/pip (python 3.13)
Looking in indexes: https://pypi.org/simple, https://wheels.home-assistant.io/musllinux-index/
Collecting questdb==2.0
  Downloading questdb-2.0.0.tar.gz (839 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 839.6/839.6 kB 67.5 MB/s eta 0:00:00
  Running command pip subprocess to install build dependencies
  Using pip 24.3.1 from /usr/local/lib/python3.13/site-packages/pip (python 3.13)
  Looking in indexes: https://pypi.org/simple, https://wheels.home-assistant.io/musllinux-index/, https://wheels.home-assistant.io/musllinux-index/
  Collecting setuptools>=45.2.0
    Downloading https://wheels.home-assistant.io/musllinux-index/setuptools-75.6.0-py3-none-any.whl (1.2 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 3.3 MB/s eta 0:00:00
  Collecting wheel>=0.34.2
    Downloading https://wheels.home-assistant.io/musllinux-index/wheel-0.45.1-py3-none-any.whl (72 kB)
  Collecting cython>=0.29.24
    Downloading https://wheels.home-assistant.io/musllinux-index/Cython-3.0.11-cp313-cp313-musllinux_1_2_x86_64.whl (3.4 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 41.1 MB/s eta 0:00:00
  Installing collected packages: wheel, setuptools, cython
    Creating /tmp/pip-build-env-o3ju0588/overlay/bin
    changing mode of /tmp/pip-build-env-o3ju0588/overlay/bin/wheel to 755
    changing mode of /tmp/pip-build-env-o3ju0588/overlay/bin/cygdb to 755
    changing mode of /tmp/pip-build-env-o3ju0588/overlay/bin/cython to 755
    changing mode of /tmp/pip-build-env-o3ju0588/overlay/bin/cythonize to 755
  Successfully installed cython-3.0.11 setuptools-75.6.0 wheel-0.45.1
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Traceback (most recent call last):
    File "/usr/local/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
      ~~~~^^
    File "/usr/local/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-o3ju0588/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=[])
             ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-o3ju0588/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
      self.run_setup()
      ~~~~~~~~~~~~~~^^
    File "/tmp/pip-build-env-o3ju0588/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 522, in run_setup
      super().run_setup(setup_script=setup_script)
      ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-o3ju0588/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 320, in run_setup
      exec(code, locals())
      ~~~~^^^^^^^^^^^^^^^^
    File "<string>", line 13, in <module>
    File "/tmp/pip-build-env-o3ju0588/overlay/lib/python3.13/site-packages/Cython/Build/__init__.py", line 1, in <module>
      from .Dependencies import cythonize
    File "/tmp/pip-build-env-o3ju0588/overlay/lib/python3.13/site-packages/Cython/Build/Dependencies.py", line 42, in <module>
      from .. import Utils
  ImportError: Error loading shared library /tmp/pip-build-env-o3ju0588/overlay/lib/python3.13/site-packages/Cython/Utils.cpython-313-x86_64-linux-musl.so: Operation not permitted
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/local/bin/python /usr/local/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpdli0_5nx
  cwd: /tmp/pip-install-hiff72si/questdb_cf88be12cc2f4578b36eaa318e06316c
  Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I confirmed that library is present when attempt to load happens, and this file has next permissions

-rwxr-xr-x    1 root     root        349056 Dec  5 17:37 /tmp/pip-build-env-kuyh8rlf/overlay/lib/python3.13/site-packages/Cython/Utils.cpython-313-x86_64-linux-musl.so

I am really out of ideas why this happens. Google search didn't help either. With 2024.11.x it has zero issues.

What version of Home Assistant Core has the issue?

core-2024.12.0

What was the last working version of Home Assistant Core?

2024.11.3

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[antst]

searching through the archive of HA Core bugs I've found that similar issue was mentioned couple of times in the past. But both times it didn't come to resolution.

[joostlek]

@cdce8p Do you have ideas?

[antst]

Unless I am wrong, it does not look like related to restrictions in the package_contrains (like possibly one of previous mentionings of similar issue).

[antst]

Could it be something like apparmor or something issue?

[antst]

OK, further info. I think it all boils down to this.

[pid 12347] stat("/tmp/pip-build-env-t1x7z2up/overlay/lib/python3.13/site-packages/Cython", {st_mode=S_IFDIR|0755, st_size=480, ...}) = 0
[pid 12347] stat("/tmp/pip-build-env-t1x7z2up/overlay/lib/python3.13/site-packages/Cython/Utils.cpython-313-x86_64-linux-musl.so", {st_mode=S_IFREG|0755, st_size=349056, ...}) = 0
[pid 12347] open("/tmp/pip-build-env-t1x7z2up/overlay/lib/python3.13/site-packages/Cython/Utils.cpython-313-x86_64-linux-musl.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[pid 12347] fcntl(3, F_SETFD, FD_CLOEXEC) = 0
[pid 12347] fstat(3, {st_mode=S_IFREG|0755, st_size=349056, ...}) = 0
[pid 12347] read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
[pid 12347] mmap(NULL, 356352, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe8b1c61000
[pid 12347] mmap(0x7fe8b1c66000, 282624, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x5000) = -1 EPERM (Operation not permitted)
[pid 12347] munmap(0x7fe8b1c61000, 356352) = 0

But after that I am lost. EPERM happens when either PROT_EXEC and file belongs to no-exec FS (not our case) or when file is sealed (not our case also). Huge pages case of EPERM even less applicable.

[antst]

Aha! Maybe it

cat /proc/mounts
...
tmpfs /tmp tmpfs rw,nosuid,nodev,noexec,relatime 0 0

EDIT:
Yes, I can confirm, that "noexec" flag on /tmp is cause of this issue.

[antst]

Although, there is another issue with QUESTDB pops up after that :)
I'd think that this is questdb issue, but taking into account that is was building before, not sure about that...checking.

[antst]

Aha, on older core, very possible, it was using prebuilt wheel.

questdb-2.0.0-cp312-cp312-musllinux_1_1_x86_64.whl

[antst]

So, results:

1. installation of questdb package (and most likely some other packages) was impossible due to /tmp mounted with noexec option, while pip toolchain was mmapping shared library with PROT_EXEC, which can not work, obviously, from "noexec" filesystem.
   In previous version of core it would be (my guess) installed from the wheel, so it would not hit this issue, or /tmp was mounted without noexec flag.
   I assume that all cython stuff, which is not available pre-built wheel for this python-musl combo, will fail.

2. there are more issues with questdb, when installed from sources, but this is irrelevant to HA Core, and resolution should continue elsewhere.

[cdce8p]

@antst I was able to build 2.0.3 from source for cp313-cp313-musllinux_1_2_x86_64. All previous versions failed for me with cython errors. Do note that only cython 3 is compatible with Python 3.13. Not sure if older versions of questdb are compatible.

[antst]

@antst I was able to build 2.0.3 from source for cp313-cp313-musllinux_1_2_x86_64. All previous versions failed for me with cython errors. Do note that only cython 3 is compatible with Python 3.13. Not sure if older versions of questdb are compatible.

I also was able to build 2.0.3. But only after remounting /tmp without noexec.
Or you mean that you will add it to the list of HA wheels?

QSS component currently depends on 1.2 (it was available as prebuilt wheels), but I already made PR which makes it compatible with 2.0.3

[cdce8p]

Or you mean that you will add it to the list of HA wheels?

It could be added to https://github.com/home-assistant/wheels-custom-integrations to build the musl wheels for it automatically. Just saw there is an open PR for it already: home-assistant/wheels-custom-integrations#705

It just needs approval.

[frenck]

I see the discussion here, however, this discussion is about a custom integration, which is not supported by the Home Assistant project.

I'm going to close this issue for that reason, as it is not actionable on our end.

../Frenck

[antst]

Actually, I still lack clarity on one question.
Is it requirement to have pre-built wheels for python packages?
Or system must be able to build python packages?

If later is valid, then there is still general issue unrelated to particular component. (mount options of /tmp)