A way to not replace system cmake

[alsrgv]

We would like to use cmake package in Horovod. One of the requirements we have is to not modify the existing CMake installed in the system.

Would it be possible to have a version of CMake package that does not place entry_points, and instead relies on the user doing:

import cmake
subprocess.check_call([os.path.join(cmake.CMAKE_BIN_DIR, 'cmake'), '...'])

?

If you have any other suggestions on how we could achieve that, they'd be much appreciated!

[jcfr]

Thanks for reaching out.

To move forward with a solution, you will find below proposal for potential solutions along with additional remarks

Potential solutions

two packages

To maintain the behavior of the current package, I am wondering if having something like this would be workable from the python packaging side:

• cmake
• cmake[without_cli_launchers]

Update:: This first approach is unlikely to work as we can't remove entrypoint upon installation of a python wheel.

We could also have there two packages but doing so would mean we change what has been made available with the current cmake python package.

• cmake
• cmake[cli_launchers]

Update: To keep the current behavior, we could install have these packages:

• cmake-binaries
• cmake

with cmake depending on cmake-binaries

This would work but involve extra maintenance work, I then wonder if the approach suggested below would be sufficient.

A script from removing extra launcher

A possibility would be provide a script removing the convenience cli launcher.

Question: Would doing something like the following be a workable solution ?

pip install cmake
python -c "import cmake; cmake.remove_cli_launchers()"

support for direct executable cmake

While convenience CLI wrappers are created in the PATH associated with the current python installation or the current virtual environment, using a script like this one is expected to work and the use the "real" cmake executable:

import os
import subprocess

import cmake

print("cmake [%s]" % os.path.join(cmake.CMAKE_BIN_DIR, 'cmake'))

subprocess.check_call([os.path.join(cmake.CMAKE_BIN_DIR, 'cmake'), '--version'])

A way to not replace system cmake

Technically, existing installation of cmake should not be replaced. After installing the wheel, new binary are made available in the PATH by creating CLI launchers in the python Scripts directory.

[alsrgv]

@jcfr, thanks for a quick response!

We noticed that when we put cmake in setup_requires= of setup.py, it does install entrypoint scripts in the /usr/local/bin, which may replace cmake that user custom-built there. Thus, removing entrypoint files post-factum may still corrupt the system.

We'd be perfectly happy to use cmake like this:

import os
import subprocess

import cmake

print("cmake [%s]" % os.path.join(cmake.CMAKE_BIN_DIR, 'cmake'))

subprocess.check_call([os.path.join(cmake.CMAKE_BIN_DIR, 'cmake'), '--version'])

I like two-package proposal (cmake-binaries that does not place entrypoints and cmake that does), but I understand the extra-overhead concern. That's an option that is most likely to work with setup_requires= though.

[jcfr]

We noticed that when we put cmake in setup_requires= of setup.py, it does install entrypoint scripts in the /usr/local/bin

Thanks for the additional details.

Thus, removing entrypoint files post-factum may still corrupt the system.

Agreed. That would not be a valid solution.

like two-package proposal (cmake-binaries that does not place entrypoints and cmake that does)

I will try to reproduce the problem and get back to you.

[alsrgv]

@jcfr, I'm wondering if you had a chance to reproduce the issue.

[henryiii]

The two package approach is common with Conda; I think it would work here.

[jcfr]

@alsrgv Thanks for your patience. Due to a limited bandwidth, I didn't have a chance to work on this yet.

To help bump the priority, an option would be to look at getting a support product. I would be happy to discuss the details offline jcfr [at] kitware [dot] com

[henryiii]

By the way, since I don't think it was mentioned here, one the main reasons this is often not a problem is that if you add cmake to your pyproject.toml file, then it is installed and used for building your package in a temporary environment, and it is never installed outside of that, so it can't interfere with anything else.

Don't use setup-requires if you can help it, pyproject.toml is superior.

[ThePrez]

@jwoehr and I are running into issues building on IBM i. Builds from source fail because it lacks custom patches included in IBM's build of CMake. We tried with -DBUILD_CMAKE_FROM_SOURCE:BOOL=OFF but the resulting behavior was that linux binaries for CMake were downloaded and placed in the cmake python distribution.

I think what we would need is "support for direct executable cmake." where the system-installed cmake is used directly, which wouldn't require the maintenance of a special cmake-binaries for this platform.
If you'd like me to open a separate issue, I can certainly do so. Thanks!

[henryiii]

One of the points of distributing a cmake package is the ability to pin CMake. I should be able to depend on cmake<=3.21 and then use features from CMake 3.21. "direct executable cmake" would have no control over this.

Wouldn't it be better to upstream the patches needed for CMake.org to build on IBM i?

[nega0]

It would be great to see some movement on this, as it absolutely effects Homebrew users who are not using a python virtual environment

❯ file $(which cmake)
/opt/homebrew/bin/cmake: Mach-O 64-bit executable arm64

❯ pip3 install --verbose --no-cache-dir cmake
Using pip 22.3.1 from /opt/homebrew/lib/python3.11/site-packages/pip (python 3.11)
Collecting cmake
  Downloading cmake-3.25.0-py2.py3-none-macosx_10_10_universal2.macosx_10_10_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl (45.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.1/45.1 MB 24.5 MB/s eta 0:00:00
Installing collected packages: cmake
  changing mode of /opt/homebrew/bin/cmake to 755
  changing mode of /opt/homebrew/bin/cpack to 755
  changing mode of /opt/homebrew/bin/ctest to 755
Successfully installed cmake-3.25.0

❯ file $(which cmake)
/opt/homebrew/bin/cmake: a /opt/homebrew/opt/[email protected]/bin/python3.11 script text executable, ASCII text

Consequently, uninstalling cmake-python-distributions leaves behind a broken CMake.

❯ pip3 uninstall cmake
Found existing installation: cmake 3.25.0
Uninstalling cmake-3.25.0:
  Would remove:
    /opt/homebrew/bin/cmake
    /opt/homebrew/bin/cpack
    /opt/homebrew/bin/ctest
    /opt/homebrew/lib/python3.11/site-packages/cmake-3.25.0.dist-info/*
    /opt/homebrew/lib/python3.11/site-packages/cmake/*
Proceed (Y/n)? y
  Successfully uninstalled cmake-3.25.0

❯ file $(which cmake)
not:   cannot open `not' (No such file or directory)
found: cannot open `found' (No such file or directory)

❯ file $(which ccmake)
/opt/homebrew/bin/ccmake: Mach-O 64-bit executable arm64

❯ brew unlink cmake && brew link cmake
Unlinking /opt/homebrew/Cellar/cmake/3.25.1... 454 symlinks removed.
Linking /opt/homebrew/Cellar/cmake/3.25.1... 457 symlinks created.

❯ file $(which cmake) $(which ccmake)
/opt/homebrew/bin/cmake:  Mach-O 64-bit executable arm64
/opt/homebrew/bin/ccmake: Mach-O 64-bit executable arm64

[henryiii]

This is installing a wheel, so there's no way this would ever change - there's no custom code run when installing a wheel. The request was for installing from SDist - this would only affect platforms without wheels. And it's probably the wrong solution, but regardless, it would literally not help the situation you describe at all.

You shouldn't install directly from PyPI into your package manager's Python pretty much ever, and this is one reason why - this directory is managed by your system package manager and should not be messed with by pip. The interaction has been improving - maybe there's a way to tell pip that the homebrew recipe blocks the installation of the pip one.

And pip on Python 3.10 & 3.11 homebrew is currently broken when not inside a venv - it can't get PEP 518 dependencies, like hatchling/flit/scikit-build(-core), etc. There is a fix in pip main, but not released yet. So if you were to try to build from source, that's busted for all packages.