Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build wheels for more platforms (Linux aarch64 wheels, MUSL wheels, etc.) #3462

Open
1 of 3 tasks
agriyakhetarpal opened this issue Oct 21, 2023 · 2 comments
Open
1 of 3 tasks
Assignees
Labels
difficulty: medium Will take a few days feature

Comments

@agriyakhetarpal
Copy link
Member

agriyakhetarpal commented Oct 21, 2023

Description

Currently, we build wheels for amd64 (Linux, macOS, Windows) and also pypy wheels on Windows (https://pypi.org/project/pybamm/#files). However, we don't provide wheels for arm64 platforms yet neither for Linux nor M-series macOS.

This means that users installing from PyPI use the source distributions, which do not contain compiled extensions. Their compilation requires pybind11 and the presence of SUNDIALS, SuiteSparse, and CasADi to link against which is easier done with an installation from source (not available and accessible for all users since this needs elevated or administrator permissions on the host system). Otherwise, pip falls back to building and storing a wheel without a platform and an ABI tag (none-any.whl) on the machine since the IDAKLU component is optional by default and doesn't compile if CMake or pybind11 are not found. This might be (slightly) easier to do after #3049 is completed.

Motivation

Easier and streamlined installation on the aforementioned platforms since the compiled extension is bundled into the wheels along with its shared external libraries (they are repaired to recreate static linkage and their runtime search paths altered)

Possible Implementation

GitHub Actions has some larger-sized M-series macOS runners now with smaller runners coming in the future. Linux arm64 runners are also on their roadmap. We can however

  1. use emulation with either QEMU (recommended by the Scientific Python guides) or Dockcross (casadi) to build arm64 wheels
  2. the Apple Xcode tools are also capable of cross-compilation to build macOS arm64 wheels in the form of a universal wheel which is double the size

Additional context

https://cibuildwheel.readthedocs.io/en/stable/faq/

Task list

@agriyakhetarpal
Copy link
Member Author

The macOS arm64 builds were successful here: https://github.com/agriyakhetarpal/PyBaMM/actions/runs/6599848898/job/17929588884#step:10:1796 through cross-compilation. I am now trying to fix troubles with not being able to find BLAS on Linux arm64. I guess openblas-devel isn't available in the index, so I have tried linking with blas-devel currently...

agriyakhetarpal added a commit to agriyakhetarpal/PyBaMM that referenced this issue Oct 23, 2023
@agriyakhetarpal agriyakhetarpal self-assigned this Oct 23, 2023
@agriyakhetarpal
Copy link
Member Author

agriyakhetarpal commented Oct 23, 2023

I was able to make some progress on this and get a BLAS distribution working using the EPEL repository for yum on CentOS 7 (on aarch64 using QEMU) on this branch. SUNDIALS can link to OpenBLAS successfully and the build-time requirements pass; however, there is a different issue now: casadi cannot be imported on pypa/manylinux2014 containers running on aarch64 architectures and returns this error during our CMake build process (we currently try importing casadi by executing the Python interpreter to get its path where it exists in the site-packages directory):

ImportError: /opt/python/cp38-cp38/lib/python3.8/site-packages/casadi/libcasadi.so.3.7:

undefined symbol: _ZTINSt6thread6_StateE

Full stack trace here: https://github.com/agriyakhetarpal/PyBaMM/actions/runs/6605214684/job/17940157625

and thus the idaklu target cannot be linked to it and compilation cannot proceed further. It is difficult to find what function or object this mangled C++ symbol represents, running objdump -T on the libcasadi.so.3.7 shared object file to analyse C++ library calls returns UND, unlike the rest of the output for the symbols.

This issue looks to be related: casadi/casadi#2887. From the discussion, it seems like it has to do with a discrepancy between the versions of the C++ compiler or a difference between the compilers that were used themselves. We would need to either use the workaround in the issue thread (I'm not sure whether it will work or whether it is even recommended for use in shared libraries since it might cause the IDAKLU extension to be non-importable with Python, even) or wait for an upstream fix. The workaround, if it should be used, could be something like:

if(CMAKE_CXX_COMPILER_ID STREQUAL "some older compiler version")
    target_link_libraries(idaklu ${SUNDIALS_LIBRARIES} casadi PRIVATE -Wl,--unresolved-symbols=ignore-in-shared-libs)
endif()

(Note: I am not super proficient with CMake so the implementation could turn out to be a bit different)

I assume this issue is the reason casadi does not use the cibuildwheel pipelines and instead uses a custom Docker image and dockcross to cross-compile for this platform. For now, I have opened #3465 to support the creation of macOS arm64 wheels which have been cross-compiled using Xcode. We can keep this issue open for a while if we wish to distribute Linux arm64 wheels in future, or close it with the linked pull request if we don't want to support this platform. In any case, installing PyBaMM from source on modern arm64 systems or via our multi-platform Docker images (where the casadi import and hence linkage works) is much easier now (cc: @martinjrobins @Saransh-cpp)

P.S. It could also be an issue with emulation using QEMU but is less likely because I can reproduce the failure in a Docker container on an M-series macOS machine. It is however possible to write another job in the workflow file that would use dockcross to do this, but it would be some work.

agriyakhetarpal added a commit to agriyakhetarpal/PyBaMM that referenced this issue Nov 27, 2023
agriyakhetarpal added a commit to agriyakhetarpal/PyBaMM that referenced this issue Nov 27, 2023
agriyakhetarpal added a commit to agriyakhetarpal/PyBaMM that referenced this issue Nov 27, 2023
kratman added a commit that referenced this issue Feb 1, 2024
…#3791)

* Add macOS M1 runner configuration for PR and scheduled tests

See #3789, #3462

* Add `macos-14` to test conditions

* Exclude Python 3.8 and 3.9 for now from testing

* Apply suggestions from code review

Co-authored-by: Eric G. Kratz <[email protected]>

* Remove some missed comments

---------

Co-authored-by: Eric G. Kratz <[email protected]>
@agriyakhetarpal agriyakhetarpal changed the title Build wheels for more platforms Build wheels for more platforms (Linux aarch64 wheels, MUSL wheels, etc.) Apr 25, 2024
js1tr3 pushed a commit to js1tr3/PyBaMM that referenced this issue Aug 12, 2024
…pybamm-team#3791)

* Add macOS M1 runner configuration for PR and scheduled tests

See pybamm-team#3789, pybamm-team#3462

* Add `macos-14` to test conditions

* Exclude Python 3.8 and 3.9 for now from testing

* Apply suggestions from code review

Co-authored-by: Eric G. Kratz <[email protected]>

* Remove some missed comments

---------

Co-authored-by: Eric G. Kratz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
difficulty: medium Will take a few days feature
Projects
None yet
Development

No branches or pull requests

1 participant