M1 Pro ARM Based Mac Installation Issue

[jack-lascala]

I am reinstalling Firedrake (and Irksome) after an automatic (and quite unfortunate) brew update of Python to 3.9.10. I am working on a M1 Pro 2021 MacBook Pro 14' and I am using the latest installation script, but it is failing. This is the relevant part of my firedrake-install.log:

creating build/lib.macosx-12-arm64-3.9/petsc4py/lib/default
  /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc -Wl,-rpath,/opt/homebrew/opt/openblas/lib -L/opt/homebrew/opt/openblas/lib -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -I/opt/homebrew/opt/openblas/include -Wno-implicit-function-declaration -g -O3 -bundle -undefined dynamic_lookup -L/Users/jack/thesis/firedrake/src/petsc/default/lib -I/Users/jack/thesis/firedrake/src/petsc/default/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk build/temp.macosx-12-arm64-3.9/default/src/PETSc.o build/temp.macosx-12-arm64-3.9/default/src/libpetsc4py.o -L/Users/jack/thesis/firedrake/src/petsc/default/lib -L/opt/homebrew/lib -L/opt/homebrew/opt/[email protected]/lib -L/opt/homebrew/opt/sqlite/lib -L/Users/jack/thesis/firedrake/src/petsc/default/lib -lpetsc -o build/lib.macosx-12-arm64-3.9/petsc4py/lib/default/PETSc.cpython-39-darwin.so
  ld: warning: dylib (/Users/jack/thesis/firedrake/src/petsc/default/lib/libmpifort.12.dylib) was built for newer macOS version (12.1) than being linked (12.0)
  ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command '/Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc' failed with exit code 1
  error: subprocess-exited-with-error
  
  × Running setup.py install for petsc4py did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

It seems that the linker cannot find the libgcc_s.1.1.dylib file, which I can see under /opt/homebrew/opt/gcc/lib/gcc/11/libgcc_s.1.1.dylib, meaning that probably the @rpath argument is not set correctly. What could I do to make sure that it is set correctly? Or could there be other fixes?

I have been suggested to ask @wence- @JDBetteridge in particular.

Many thanks!

Giacomo

[wence-]

ld: warning: dylib (/Users/jack/thesis/firedrake/src/petsc/default/lib/libmpifort.12.dylib) was built for newer macOS version (12.1) than being linked (12.0)
ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc' failed with exit code 1
error: subprocess-exited-with-error

I suppose that this is from scratch new install. We're far enough through at this point that we should be able to debug just the failing bits.

Can you activate the virtual environment, do

export MACOSX_DEPLOYMENT_TARGET=12.1
export PETSC_DIR=/Users/jack/thesis/firedrake/src/petsc
export PETSC_ARCH=default
cd $PETSC_DIR
pip install -U --no-deps src/binding/petsc4py

and show the output?

[jack-lascala]

Here it is. It looks like the same error as before repeated twice, once for setup.py bdist_wheel and once for setup.py for petsc4py.

Processing ./src/binding/petsc4py
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: petsc4py
  Building wheel for petsc4py (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      running bdist_wheel
      running build
      running build_src
      running build_py
      running build_ext
      PETSC_DIR:    /Users/jack/thesis/firedrake/src/petsc
      PETSC_ARCH:   default
      version:      3.15.4 development
      integer-size: 32-bit
      scalar-type:  real
      precision:    double
      language:     CONLY
      compiler:     /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc
      linker:       /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc
      building 'PETSc' extension
      /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc -I/opt/homebrew/opt/openblas/include -Wno-implicit-function-declaration -g -O3 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/jack/thesis/firedrake/src/petsc/default/include/eigen3 -I/Users/jack/thesis/firedrake/src/petsc/default/include -I/Users/jack/thesis/firedrake/src/petsc/include -Isrc/include -I/Users/jack/thesis/firedrake/lib/python3.9/site-packages/numpy/core/include -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I/opt/homebrew/opt/sqlite/include -I/Users/jack/thesis/firedrake/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/PETSc.c -o build/temp.macosx-12-arm64-3.9/default/src/PETSc.o
      /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc -I/opt/homebrew/opt/openblas/include -Wno-implicit-function-declaration -g -O3 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/jack/thesis/firedrake/src/petsc/default/include/eigen3 -I/Users/jack/thesis/firedrake/src/petsc/default/include -I/Users/jack/thesis/firedrake/src/petsc/include -Isrc/include -I/Users/jack/thesis/firedrake/lib/python3.9/site-packages/numpy/core/include -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I/opt/homebrew/opt/sqlite/include -I/Users/jack/thesis/firedrake/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/libpetsc4py.c -o build/temp.macosx-12-arm64-3.9/default/src/libpetsc4py.o
      /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc -Wl,-rpath,/opt/homebrew/opt/openblas/lib -L/opt/homebrew/opt/openblas/lib -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -I/opt/homebrew/opt/openblas/include -Wno-implicit-function-declaration -g -O3 -bundle -undefined dynamic_lookup -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk build/temp.macosx-12-arm64-3.9/default/src/PETSc.o build/temp.macosx-12-arm64-3.9/default/src/libpetsc4py.o -L/Users/jack/thesis/firedrake/src/petsc/default/lib -L/opt/homebrew/lib -L/opt/homebrew/opt/[email protected]/lib -L/opt/homebrew/opt/sqlite/lib -L/Users/jack/thesis/firedrake/src/petsc/default/lib -lpetsc -o build/lib.macosx-12-arm64-3.9/petsc4py/lib/default/PETSc.cpython-39-darwin.so
      ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      error: command '/Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for petsc4py
  Running setup.py clean for petsc4py
Failed to build petsc4py
Installing collected packages: petsc4py
  Running setup.py install for petsc4py ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for petsc4py did not run successfully.
  │ exit code: 1
  ╰─> [47 lines of output]
      running install
      /Users/jack/thesis/firedrake/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_src
      running build_py
      creating build
      creating build/lib.macosx-12-arm64-3.9
      creating build/lib.macosx-12-arm64-3.9/petsc4py
      copying src/PETSc.py -> build/lib.macosx-12-arm64-3.9/petsc4py
      copying src/__init__.py -> build/lib.macosx-12-arm64-3.9/petsc4py
      copying src/__main__.py -> build/lib.macosx-12-arm64-3.9/petsc4py
      creating build/lib.macosx-12-arm64-3.9/petsc4py/lib
      copying src/lib/__init__.py -> build/lib.macosx-12-arm64-3.9/petsc4py/lib
      creating build/lib.macosx-12-arm64-3.9/petsc4py/include
      creating build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/include/petsc4py/numpy.h -> build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/include/petsc4py/petsc4py.h -> build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/include/petsc4py/petsc4py.PETSc.h -> build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/include/petsc4py/petsc4py.PETSc_api.h -> build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/include/petsc4py/petsc4py.i -> build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/include/petsc4py/PETSc.pxd -> build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/include/petsc4py/__init__.pxd -> build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/include/petsc4py/__init__.pyx -> build/lib.macosx-12-arm64-3.9/petsc4py/include/petsc4py
      copying src/PETSc.pxd -> build/lib.macosx-12-arm64-3.9/petsc4py
      copying src/lib/petsc.cfg -> build/lib.macosx-12-arm64-3.9/petsc4py/lib
      running build_ext
      PETSC_DIR:    /Users/jack/thesis/firedrake/src/petsc
      PETSC_ARCH:   default
      version:      3.15.4 development
      integer-size: 32-bit
      scalar-type:  real
      precision:    double
      language:     CONLY
      compiler:     /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc
      linker:       /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc
      building 'PETSc' extension
      creating build/temp.macosx-12-arm64-3.9
      creating build/temp.macosx-12-arm64-3.9/default
      creating build/temp.macosx-12-arm64-3.9/default/src
      /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc -I/opt/homebrew/opt/openblas/include -Wno-implicit-function-declaration -g -O3 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/jack/thesis/firedrake/src/petsc/default/include/eigen3 -I/Users/jack/thesis/firedrake/src/petsc/default/include -I/Users/jack/thesis/firedrake/src/petsc/include -Isrc/include -I/Users/jack/thesis/firedrake/lib/python3.9/site-packages/numpy/core/include -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I/opt/homebrew/opt/sqlite/include -I/Users/jack/thesis/firedrake/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/PETSc.c -o build/temp.macosx-12-arm64-3.9/default/src/PETSc.o
      /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc -I/opt/homebrew/opt/openblas/include -Wno-implicit-function-declaration -g -O3 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/jack/thesis/firedrake/src/petsc/default/include/eigen3 -I/Users/jack/thesis/firedrake/src/petsc/default/include -I/Users/jack/thesis/firedrake/src/petsc/include -Isrc/include -I/Users/jack/thesis/firedrake/lib/python3.9/site-packages/numpy/core/include -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I/opt/homebrew/opt/sqlite/include -I/Users/jack/thesis/firedrake/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/libpetsc4py.c -o build/temp.macosx-12-arm64-3.9/default/src/libpetsc4py.o
      creating build/lib.macosx-12-arm64-3.9/petsc4py/lib/default
      /Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc -Wl,-rpath,/opt/homebrew/opt/openblas/lib -L/opt/homebrew/opt/openblas/lib -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -I/opt/homebrew/opt/openblas/include -Wno-implicit-function-declaration -g -O3 -bundle -undefined dynamic_lookup -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk build/temp.macosx-12-arm64-3.9/default/src/PETSc.o build/temp.macosx-12-arm64-3.9/default/src/libpetsc4py.o -L/Users/jack/thesis/firedrake/src/petsc/default/lib -L/opt/homebrew/lib -L/opt/homebrew/opt/[email protected]/lib -L/opt/homebrew/opt/sqlite/lib -L/Users/jack/thesis/firedrake/src/petsc/default/lib -lpetsc -o build/lib.macosx-12-arm64-3.9/petsc4py/lib/default/PETSc.cpython-39-darwin.so
      ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      error: command '/Users/jack/thesis/firedrake/src/petsc/default/bin/mpicc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> petsc4py

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[jack-lascala]

I've seen that Issue #2320 discusses what seems to be the same exact problem, apologies for not seeing this earlier and duplicating. I can try resolving the issue in the ways suggested there for the meantime.

[jack-lascala]

I can build and install Firedrake using the latest install script and running export LDFLAGS="-L/opt/homebrew/opt/gcc/lib/gcc/11" just before it, although I also experience no tests passing like in #2320. I've tried running the "basic Helmholtz equation" tutorial as well but any invocation to mpicc fails because of not finding @rpath/libgcc_s.1.1.dylib.

[jack-lascala]

I have managed to complete a full installation after having created a symbolic link to my real libgcc_s.1.1.dylib (living in /opt/homebrew/opt/gcc/lib/gcc/11/) from the /opt/homebrew/opt/openblas/lib directory, which is set as the rpath passed to the linker in the install log. I am failing only 3/27 of the suggested tests. I also can now successfully run some examples/tutorials like the basic Helmholtz.

It looks like maybe the command assembled by the installation script should not be looking for libgcc inside the openblas homebrew directory.

[dham]

So it turns out that adding -L/opt/homebrew/opt/gcc/lib/gcc/11 to the mpicc command line also makes this better. I need to work out how to make that automatically happen. This is probably a better option than having users make symlinks.

[jack-lascala]

Update: installation of Firedrake with a symlink of libgcc_s.1.1.dylib in /opt/homebrew/opt/openblas/lib is successful and running the suggested basic tests with pytest==6.2.5 results in 27/27 tests passing.

[wence-]

Thanks. This is useful information. It feels like it should not be necessary, but st the moment none of the development team have an M1 system, so figuring things out is rather trial and error.

[dham]

I believe that #2408 fixed this. M1 users who have an issue after this point, please open a new discussion.