MacOS: Segmentation Fault 11 When Importing C++ Modules

[rbiessel]

Hi All,

I'm getting a segmentation fault whenever the c++ modules are imported.

(fringe) MBP:fringe rbiessel$ python
Python 3.7.7 (default, Mar 23 2020, 17:31:31) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nmaplib
Segmentation fault: 11

The same thing happens when running nmap.py. Has anyone else encountered this? Is this likely an installation problem? I've tried reinstalling the package without any luck.

Here's the build/installation process I followed according to the installation instructions:

cmake

:build rbiessel$ CXX=${CXX} cmake -DCMAKE_INSTALL_PREFIX=../install ../src/fringe -DCMAKE_FIND_FRAMEWORK=NEVER
-- The C compiler identification is Clang 10.0.0
-- The CXX compiler identification is Clang 10.0.0
-- Check for working C compiler: /opt/anaconda3/envs/fringe/bin/x86_64-apple-darwin13.4.0-clang
-- Check for working C compiler: /opt/anaconda3/envs/fringe/bin/x86_64-apple-darwin13.4.0-clang -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/anaconda3/envs/fringe/bin/x86_64-apple-darwin13.4.0-clang++
-- Check for working CXX compiler: /opt/anaconda3/envs/fringe/bin/x86_64-apple-darwin13.4.0-clang++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found GDAL: /opt/anaconda3/envs/fringe/lib/libgdal.dylib (found suitable version "3.0.2", minimum required is "3.0") 
-- Found Armadillo: /opt/local/lib/libarmadillo.dylib (found suitable version "10.3.0", minimum required is "8.100") 
-- Found OpenMP_CXX: -fopenmp=libomp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5") found components:  CXX 
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Found BLAS: /opt/anaconda3/envs/fringe/lib/libmkl_intel_lp64.dylib;/opt/anaconda3/envs/fringe/lib/libmkl_intel_thread.dylib;/opt/anaconda3/envs/fringe/lib/libmkl_core.dylib;/opt/anaconda3/envs/fringe/lib/libiomp5.dylib;-lm;-ldl  
-- Looking for cheev_
-- Looking for cheev_ - found
-- A library with LAPACK API found.
-- Found Python: /opt/anaconda3/envs/fringe/bin/python3.7 (found suitable version "3.7.7", minimum required is "3.7") found components:  Interpreter Development NumPy 
-- Found Python: /opt/anaconda3/envs/fringe/bin/python3.7 (found suitable version "3.7.7", minimum required is "3.6") found components:  Interpreter 
-- Found Cython: /opt/local/bin/cython-3.7  
-- Found Python: /opt/anaconda3/envs/fringe/lib/libpython3.7m.dylib (found version "3.7.7") found components:  Development 
-- Found Cython:  0.29.22
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/rbiessel/Documents/apps/ISCE/fringe/build

make all

:build rbiessel$ make all
[  3%] Compiling Cython CXX source for calamplib...
Scanning dependencies of target calamplib
[  7%] Building CXX object src/calamp/CMakeFiles/calamplib.dir/calamplib.cxx.o
[ 10%] Linking CXX shared module calamplib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 10%] Built target calamplib
[ 14%] Compiling Cython CXX source for ampdispersionlib...
Scanning dependencies of target ampdispersionlib
[ 17%] Building CXX object src/ampdispersion/CMakeFiles/ampdispersionlib.dir/ampdispersionlib.cxx.o
[ 21%] Linking CXX shared module ampdispersionlib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 21%] Built target ampdispersionlib
[ 25%] Compiling Cython CXX source for nmaplib...
Scanning dependencies of target nmaplib
[ 28%] Building CXX object src/nmap/CMakeFiles/nmaplib.dir/nmaplib.cxx.o
[ 32%] Linking CXX shared module nmaplib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 32%] Built target nmaplib
[ 35%] Compiling Cython CXX source for despecklib...
Scanning dependencies of target despecklib
[ 39%] Building CXX object src/despeck/CMakeFiles/despecklib.dir/despecklib.cxx.o
[ 42%] Linking CXX shared module despecklib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 42%] Built target despecklib
[ 46%] Compiling Cython CXX source for evdlib...
Scanning dependencies of target evdlib
[ 50%] Building CXX object src/evd/CMakeFiles/evdlib.dir/evdlib.cxx.o
[ 53%] Linking CXX shared module evdlib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 53%] Built target evdlib
[ 57%] Compiling Cython CXX source for sbaslib...
Scanning dependencies of target sbaslib
[ 60%] Building CXX object src/sbas/CMakeFiles/sbaslib.dir/sbaslib.cxx.o
[ 64%] Linking CXX shared module sbaslib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 64%] Built target sbaslib
Scanning dependencies of target fit
[ 67%] Building CXX object src/fit/CMakeFiles/fit.dir/Fit.cc.o
[ 71%] Linking CXX shared library libfit.dylib
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 71%] Built target fit
[ 75%] Compiling Cython CXX source for fitlib...
Scanning dependencies of target fitlib
[ 78%] Building CXX object src/fit/CMakeFiles/fitlib.dir/fitlib.cxx.o
[ 82%] Linking CXX shared module fitlib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 82%] Built target fitlib
Scanning dependencies of target unwrap
[ 85%] Building CXX object src/unwrap_error/CMakeFiles/unwrap.dir/Unwrap.cc.o
[ 89%] Linking CXX shared library libunwrap.dylib
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 89%] Built target unwrap
[ 92%] Compiling Cython CXX source for unwraplib...
Scanning dependencies of target unwraplib
[ 96%] Building CXX object src/unwrap_error/CMakeFiles/unwraplib.dir/unwraplib.cxx.o
[100%] Linking CXX shared module unwraplib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[100%] Built target unwraplib

make install

:build rbiessel$ make install
[ 10%] Built target calamplib
[ 21%] Built target ampdispersionlib
[ 32%] Built target nmaplib
[ 42%] Built target despecklib
[ 53%] Built target evdlib
[ 64%] Built target sbaslib
[ 71%] Built target fit
[ 82%] Built target fitlib
[ 89%] Built target unwrap
[100%] Built target unwraplib
Install the project...
-- Install configuration: ""
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/calamplib.so
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/calamp.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/ampdispersionlib.so
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/ampdispersion.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/nmaplib.so
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/nmap.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/despecklib.so
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/despeck.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/evdlib.so
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/evd.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/Stack.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/sequential.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/sbaslib.so
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/sbas.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/lib/libfit.dylib
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/fitlib.so
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/TimeFunction.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/fit_timeseries.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/BasisFn.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/CenteredBasisFn.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Collection.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Constant.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Exp.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Interface.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Log.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Periodic.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Polynomial.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Power.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Spline.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/SplineFactory.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Trigonometry.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/__init__.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/datetimeUtils.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/lib/libunwrap.dylib
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/unwraplib.so
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/uavsar2vrt.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/stripmap2vrt.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/tops2vrt.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/adjustMiniStacks.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/unwrap_fringe.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/unwrapStack.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/Network.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/StackVRT.py
-- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/integratePS.py

[rbiessel]

Hi, @hfattahi / @piyushrpt, I spent some more time trying to get around this issue and I was wondering if either of you had any insight? I'm also hoping to contribute code eventually. Do either of you have any suggested resources pertaining development workflows via MacOS?

[piyushrpt]

new macs are a little weird. gcc is pretty much unusable. Use clang.
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX={installdir} {srcdir}

[rbiessel]

new macs are a little weird. gcc is pretty much unusable. Use clang.
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX={installdir} {srcdir}

I've been using clang to build FRInGE from the start but I have tried gcc as well. Clang does seem to get farther in the process but I'm still getting a segmentation fault. This time with Clang 11.0.0:

Python 3.7.8 | packaged by conda-forge | (default, Nov 27 2020, 19:11:24) 
[Clang 11.0.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nmaplib
Segmentation fault: 11

Do you think this ld: warning is the issue?

[ 25%] Compiling Cython CXX source for nmaplib...
Scanning dependencies of target nmaplib
[ 28%] Building CXX object src/nmap/CMakeFiles/nmaplib.dir/nmaplib.cxx.o
[ 32%] Linking CXX shared module nmaplib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 32%] Built target nmaplib

[piyushrpt]

clang-11 and gcc-10 are both a little weird as they get stricter with what can be done. I have only tested cmake with clang 9/10. Maybe this will help - https://andreasfertig.blog/2021/02/clang-and-gcc-on-macos-catalina-finding-the-include-paths/

[rbiessel]

Building with clang 10 (the mp part seemed necessary, otherwise cmake couldn't find OpenMP)

rbiessel$ CXX=clang++-mp-10 CC=clang-mp-10 cmake -DCMAKE_FIND_FRAMEWORK=NEVER -DCMAKE_INSTALL_PREFIX=../install ../src/fringe

Python 3.7.8 | packaged by conda-forge | (default, Nov 27 2020, 19:11:24) 
[Clang 11.0.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nmaplib
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/rbiessel/Documents/apps/ISCE/fringe/install/python/nmaplib.so, 2): Symbol not found: _PyObject_GC_IsFinalized
  Referenced from: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/nmaplib.so
  Expected in: flat namespace
 in /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/nmaplib.so

[piyushrpt]

Can you post the output of cmake - this looks like the right python is not being used / discovered

[rbiessel]

I was reading this kind of error could be caused by packages compiled with conflicting libraries?

Might it be because it's finding packages outside of the conda environment?

(fringe) Rowans-MBP:build rbiessel$ CXX=clang++-mp-10 CC=clang-mp-10 cmake -DCMAKE_PREFIX_PATH=$CONDA_PREFIX  -DCMAKE_INSTALL_PREFIX=../install ../src/fringe
-- The C compiler identification is Clang 10.0.1
-- The CXX compiler identification is Clang 10.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/local/bin/clang-mp-10 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/local/bin/clang++-mp-10 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found GDAL: /opt/anaconda3/envs/fringe/lib/libgdal.dylib (found suitable version "3.0.4", minimum required is "3.0") 
-- Found Armadillo: /opt/anaconda3/envs/fringe/include (found suitable version "9.900.4", minimum required is "8.100") 
-- Found OpenMP_CXX: -fopenmp=libomp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5") found components: CXX 
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Found BLAS: /opt/local/lib/libopenblas.dylib  
-- Looking for cheev_
-- Looking for cheev_ - found
-- Found LAPACK: /opt/local/lib/libopenblas.dylib;-lm;-ldl  
-- Found Python: /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 (found suitable version "3.9.5", minimum required is "3.7") found components: Interpreter Development NumPy Development.Module Development.Embed 
-- Found Python: /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 (found suitable version "3.9.5", minimum required is "3.6") found components: Interpreter 
-- Found Cython: /opt/local/bin/cython-3.9  
-- Found Python: /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 (found version "3.9.5") found components: Development Development.Module Development.Embed 
-- Found Cython:  0.29.22
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/rbiessel/Documents/apps/ISCE/fringe/build
(fringe) Row

[rbiessel]

Actually, I got nmaplib.py (and other modules) to import succesfully using python3.9 (instead of the conda enviromnent's python3.7)! I will now see if I can use the workflow and if it will actually work! This is the farthest I've gotten with it so far on this macbook though. Thank you for your help Piyush

[piyushrpt]

Use -DPython3_EXECUTABLE=path-to-python3-you-want during cmake to make it specifically use that version.
Or use -DCMAKE_FIND_FRAMEWORK=NEVER to use python3 that is active in the environment.