Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix dependencies - get from conda #14

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ add_executable(${XTENSOR_BENCHMARK_TARGET} ${XTENSOR_BENCHMARK} ${XTENSOR_HEADER

# Mandatory dependencies
# ======================

find_package(xtensor REQUIRED CONFIG)
find_package(xsimd REQUIRED)
find_package(xtl REQUIRED)
Expand Down Expand Up @@ -135,8 +134,8 @@ endif()

if(BENCHMARK_PYTHONIC)
find_package(PythonLibs)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in conda env, the cmake is still picking wrong python (that may be the reason for Pythran, numpy errors?) :

env:

 $ which python
/home/mmm/.conda/envs/xtensor-benchmark/bin/python
(xtensor-benchmark) mmm@mmm-U2442:~/devel/xtensor-benchmark/build$ python --version
Python 3.7.1

while in cmake (3.8!!):

-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2")
-- Found PythonInterp: /usr/bin/python3.8 (found version "3.8.2")

I guess it's with priority of the paths?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

found relevant issue, usnistgov/feasst#2
but even after applying that solution, still detects wrong python

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really don't get it, I've just tested your branch on my machine and it could detect everything correctly. I'm using conda 4.8.3 (from miniconda) on gentoo.

Copy link
Author

@breznak breznak Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my complete cmake output (for all benchmarks)


(xtensor-benchmark) mmm@mmm-U2442:~/devel/xtensor-benchmark$ cmake . -DBENCHMARK_ALL=ON
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Forcing tests build type to Release
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found Blitz: /home/mmm/devel/.conda/envs/xtensor-benchmark/include  
-- Found Armadillo: /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so (found version "9.200.7") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2") 
-- Found PythonInterp: /usr/bin/python3.8 (found version "3.8.2") 
CMake Error at modules/FindNumpy.cmake:68 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:69 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:70 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:71 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:74 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:77 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:81 (string):
string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty
string.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:82 (math):
math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) +
NOTFOUND": syntax error, unexpected exp_TIMES (11).
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindPythran.cmake:43 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:44 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:45 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:46 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:49 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:52 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:56 (string):
string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty
string.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:57 (math):
math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) +
NOTFOUND": syntax error, unexpected exp_TIMES (11).
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)




COMPILING WITH
======================================


COMPILER        : /usr/bin/c++
FLAGS           :  -Ofast -ffast-math -march=native -pthread

Found xtensor   : /home/mmm/.conda/envs/xtensor-benchmark/include
Found xsimd     : /home/mmm/.conda/envs/xtensor-benchmark/include


Found eigen     : /home/mmm/.conda/envs/xtensor-benchmark/include/eigen3
Found Blitz     : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libblitz.so
Found Armadillo : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so
Found Pythran   :
Using benchmark :  | benchmark::benchmark_main
-- Configuring incomplete, errors occurred!
See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeOutput.log".
See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeError.log".

I'm using conda 4.8.3 (from miniconda) on gentoo.

I'm also on conda 4.8.3, Ubuntu 20.04 LTS.
The use of system python would be the main cause, then the other conda-installed libs get missing.

My cmake is 3.16.0

Copy link
Member

@JohanMabille JohanMabille Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you pass -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX to the cmake command? (and check that it is correctly set in CMakeCache.txt efter running cmake).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure.
The err is the same.
Setup:

git clean -xdf
cmake . -DBENCHMARK_ALL=ON -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX 
echo $CONDA_PREFIX
> /home/mmm/.conda/envs/xtensor-benchmark 

#validate
$ grep 'PREFIX' CMakeCache.txt
CMAKE_INSTALL_PREFIX:PATH=/home/mmm/.conda/envs/xtensor-benchmark

Cmake Output:

$ echo $CONDA_PREFIX
/home/mmm/.conda/envs/xtensor-benchmark
(xtensor-benchmark) mmm@mmm-U2442:~/devel/xtensor-benchmark$ cmake . -DBENCHMARK_ALL=ON -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX 
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Forcing tests build type to Release
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found Blitz: /home/mmm/devel/.conda/envs/xtensor-benchmark/include  
-- Found Armadillo: /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so (found version "9.200.7") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2") 
-- Found PythonInterp: /usr/bin/python3.8 (found version "3.8.2") 
CMake Error at modules/FindNumpy.cmake:68 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:69 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:70 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:71 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:74 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:77 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:81 (string):
string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty
string.
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindNumpy.cmake:82 (math):
math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) +
NOTFOUND": syntax error, unexpected exp_TIMES (11).
Call Stack (most recent call first):
CMakeLists.txt:137 (find_package)


CMake Error at modules/FindPythran.cmake:43 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:44 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:45 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:46 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:49 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:52 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:56 (string):
string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty
string.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:57 (math):
math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) +
NOTFOUND": syntax error, unexpected exp_TIMES (11).
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)




COMPILING WITH
======================================


COMPILER        : /usr/bin/c++
FLAGS           :  -Ofast -ffast-math -march=native -pthread

Found xtensor   : /home/mmm/.conda/envs/xtensor-benchmark/include
Found xsimd     : /home/mmm/.conda/envs/xtensor-benchmark/include


Found eigen     : /home/mmm/.conda/envs/xtensor-benchmark/include/eigen3
Found Blitz     : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libblitz.so
Found Armadillo : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so
Found Pythran   :
Using benchmark :  | benchmark::benchmark_main
-- Configuring incomplete, errors occurred!
See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeOutput.log".
See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeError.log".

Copy link
Member

@JohanMabille JohanMabille Jun 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We got a hint for a similar problem with xtensor-python, can you pass the -DPYTHON_EXECUTABLE:FILEPATH=path_to_python to the cmake command line and see if it fixes the issue?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JohanMabille , I guess it helps a bit - thanks to that it: finds NumPy, uses correct Python (3.7), BUT python libs now mismatch (3.8)! Is there a similar setting for pythonlibs?

CMake output below

$ git clean -xdf
Removing CMakeCache.txt
Removing CMakeFiles/
(xtensor-benchmark) mmm@mmm-U2442:~/devel/xtensor-benchmark$ cmake . -DBENCHMARK_ALL=ON -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DPYTHON_EXECUTABLE:FILEPATH=`which python` 
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Forcing tests build type to Release
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found Blitz: /home/mmm/devel/.conda/envs/xtensor-benchmark/include  
-- Found Armadillo: /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so (found version "9.200.7") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2") 
-- Found PythonInterp: /home/mmm/.conda/envs/xtensor-benchmark/bin/python (found version "3.7.1") 
-- Found NumPy: version "1.18.5" /home/mmm/.conda/envs/xtensor-benchmark/lib/python3.7/site-packages/numpy/core/include
CMake Error at modules/FindPythran.cmake:43 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:44 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:45 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:46 (list):
list GET given empty list
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:49 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:52 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:56 (string):
string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty
string.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)


CMake Error at modules/FindPythran.cmake:57 (math):
math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) +
NOTFOUND": syntax error, unexpected exp_TIMES (11).
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)




COMPILING WITH
======================================


COMPILER        : /usr/bin/c++
FLAGS           :  -Ofast -ffast-math -march=native -pthread

Found xtensor   : /home/mmm/.conda/envs/xtensor-benchmark/include
Found xsimd     : /home/mmm/.conda/envs/xtensor-benchmark/include


Found eigen     : /home/mmm/.conda/envs/xtensor-benchmark/include/eigen3
Found Blitz     : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libblitz.so
Found Armadillo : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so
Found Pythran   :
Using benchmark :  | benchmark::benchmark_main
-- Configuring incomplete, errors occurred!
See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeOutput.log".
See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeError.log".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like you can pass a -DPYTHON_LIBRARIES=path_to_python_lib option (or DPYTHON_LIBRARY for very olf version of cmake, i.e. < 3) to cmake. Let's hope it fixes the problem for good!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, no. We're able to set the correct pythonlibs path, but it is still somehow overriden by python 3.8?!

-- Found Blitz: /home/mmm/devel/.conda/envs/xtensor-benchmark/include  
-- Found Armadillo: /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so (found version "9.200.7") 
-- Found PythonLibs: /home/mmm/.conda/envs/xtensor-benchmark/lib/python3.7 (found version "3.8.2") 
-- Found PythonInterp: /home/mmm/.conda/envs/xtensor-benchmark/bin/python (found version "3.7.1") 
-- Found NumPy: version "1.18.5" /home/mmm/.conda/envs/xtensor-benchmark/lib/python3.7/site-packages/numpy/core/include
CMake Error at modules/FindPythran.cmake:43 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:138 (find_package)

I tried to go the other way and install py3.8, but conda env has some unresolvable dependency conflicts and is unable to do that.

find_package(NumPy)
find_package(Pythran)
find_package(Numpy)
find_package(Pythran )
link_directories(${Pythran_INCLUDE_DIRS})
target_compile_definitions(${XTENSOR_BENCHMARK_TARGET} PRIVATE HAS_PYTHONIC=1 ENABLE_PYTHON_MODULE=1 USE_BOOST_SIMD=1)
target_include_directories(${XTENSOR_BENCHMARK_TARGET} PRIVATE
Expand Down
11 changes: 6 additions & 5 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,15 @@ channels:
- conda-forge
- defaults
dependencies:
- eigen
- conda-forge::benchmark
- conda-forge::libblitz
- conda-forge::pythran
- eigen>=3.0
- armadillo
- xsimd
- xtensor
- xtl
- cmake
- cmake>=3.1
- gmp
- numpy
- python
- pip:
- pythran
- python>=3.0