Fix dependencies - get from conda #14

breznak · 2020-06-05T00:05:17Z

fixes some errors in cmake, add dependencies.

For #13

not from pip

JohanMabille

THanks a lot for tackling this! It has been on our todo list for a while but we didn't find the time to do it.

environment.yml

CMakeLists.txt

typo in package name, thanks Johan!

This reverts commit 1ab19c4.

breznak

I can now compile & run the basic (xtensor) tests!
But for other frameworks, there are still errors, when compiling with
-DBENCHMARK_ALL=ON

breznak · 2020-06-05T08:21:27Z

CMakeLists.txt

@@ -135,8 +134,8 @@ endif()

 if(BENCHMARK_PYTHONIC)
    find_package(PythonLibs)


in conda env, the cmake is still picking wrong python (that may be the reason for Pythran, numpy errors?) :

env:

$ which python /home/mmm/.conda/envs/xtensor-benchmark/bin/python (xtensor-benchmark) mmm@mmm-U2442:~/devel/xtensor-benchmark/build$ python --version Python 3.7.1

while in cmake (3.8!!):

-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2")
-- Found PythonInterp: /usr/bin/python3.8 (found version "3.8.2")

I guess it's with priority of the paths?

found relevant issue, usnistgov/feasst#2
but even after applying that solution, still detects wrong python

I really don't get it, I've just tested your branch on my machine and it could detect everything correctly. I'm using conda 4.8.3 (from miniconda) on gentoo.

This is my complete cmake output (for all benchmarks)

(xtensor-benchmark) mmm@mmm-U2442:~/devel/xtensor-benchmark$ cmake . -DBENCHMARK_ALL=ON -- The C compiler identification is GNU 9.3.0 -- The CXX compiler identification is GNU 9.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Forcing tests build type to Release -- Performing Test HAS_CPP14_FLAG -- Performing Test HAS_CPP14_FLAG - Success -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Found Blitz: /home/mmm/devel/.conda/envs/xtensor-benchmark/include -- Found Armadillo: /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so (found version "9.200.7") -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2") -- Found PythonInterp: /usr/bin/python3.8 (found version "3.8.2") CMake Error at modules/FindNumpy.cmake:68 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:69 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:70 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:71 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:74 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:77 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:81 (string): string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty string. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:82 (math): math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) + NOTFOUND": syntax error, unexpected exp_TIMES (11). Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindPythran.cmake:43 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:44 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:45 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:46 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:49 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:52 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:56 (string): string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty string. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:57 (math): math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) + NOTFOUND": syntax error, unexpected exp_TIMES (11). Call Stack (most recent call first): CMakeLists.txt:138 (find_package) COMPILING WITH ====================================== COMPILER : /usr/bin/c++ FLAGS : -Ofast -ffast-math -march=native -pthread Found xtensor : /home/mmm/.conda/envs/xtensor-benchmark/include Found xsimd : /home/mmm/.conda/envs/xtensor-benchmark/include Found eigen : /home/mmm/.conda/envs/xtensor-benchmark/include/eigen3 Found Blitz : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libblitz.so Found Armadillo : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so Found Pythran : Using benchmark : | benchmark::benchmark_main -- Configuring incomplete, errors occurred! See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeOutput.log". See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeError.log".

I'm using conda 4.8.3 (from miniconda) on gentoo.

I'm also on conda 4.8.3, Ubuntu 20.04 LTS.
The use of system python would be the main cause, then the other conda-installed libs get missing.

My cmake is 3.16.0

can you pass -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX to the cmake command? (and check that it is correctly set in CMakeCache.txt efter running cmake).

sure.
The err is the same.
Setup:

git clean -xdf cmake . -DBENCHMARK_ALL=ON -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX echo $CONDA_PREFIX > /home/mmm/.conda/envs/xtensor-benchmark #validate $ grep 'PREFIX' CMakeCache.txt CMAKE_INSTALL_PREFIX:PATH=/home/mmm/.conda/envs/xtensor-benchmark

Cmake Output:

$ echo $CONDA_PREFIX /home/mmm/.conda/envs/xtensor-benchmark (xtensor-benchmark) mmm@mmm-U2442:~/devel/xtensor-benchmark$ cmake . -DBENCHMARK_ALL=ON -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -- The C compiler identification is GNU 9.3.0 -- The CXX compiler identification is GNU 9.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Forcing tests build type to Release -- Performing Test HAS_CPP14_FLAG -- Performing Test HAS_CPP14_FLAG - Success -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Found Blitz: /home/mmm/devel/.conda/envs/xtensor-benchmark/include -- Found Armadillo: /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so (found version "9.200.7") -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2") -- Found PythonInterp: /usr/bin/python3.8 (found version "3.8.2") CMake Error at modules/FindNumpy.cmake:68 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:69 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:70 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:71 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:74 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:77 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:81 (string): string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty string. Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindNumpy.cmake:82 (math): math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) + NOTFOUND": syntax error, unexpected exp_TIMES (11). Call Stack (most recent call first): CMakeLists.txt:137 (find_package) CMake Error at modules/FindPythran.cmake:43 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:44 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:45 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:46 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:49 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:52 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:56 (string): string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty string. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:57 (math): math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) + NOTFOUND": syntax error, unexpected exp_TIMES (11). Call Stack (most recent call first): CMakeLists.txt:138 (find_package) COMPILING WITH ====================================== COMPILER : /usr/bin/c++ FLAGS : -Ofast -ffast-math -march=native -pthread Found xtensor : /home/mmm/.conda/envs/xtensor-benchmark/include Found xsimd : /home/mmm/.conda/envs/xtensor-benchmark/include Found eigen : /home/mmm/.conda/envs/xtensor-benchmark/include/eigen3 Found Blitz : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libblitz.so Found Armadillo : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so Found Pythran : Using benchmark : | benchmark::benchmark_main -- Configuring incomplete, errors occurred! See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeOutput.log". See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeError.log".

We got a hint for a similar problem with xtensor-python, can you pass the -DPYTHON_EXECUTABLE:FILEPATH=path_to_python to the cmake command line and see if it fixes the issue?

Thanks @JohanMabille , I guess it helps a bit - thanks to that it: finds NumPy, uses correct Python (3.7), BUT python libs now mismatch (3.8)! Is there a similar setting for pythonlibs?

CMake output below

$ git clean -xdf Removing CMakeCache.txt Removing CMakeFiles/ (xtensor-benchmark) mmm@mmm-U2442:~/devel/xtensor-benchmark$ cmake . -DBENCHMARK_ALL=ON -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DPYTHON_EXECUTABLE:FILEPATH=`which python` -- The C compiler identification is GNU 9.3.0 -- The CXX compiler identification is GNU 9.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Forcing tests build type to Release -- Performing Test HAS_CPP14_FLAG -- Performing Test HAS_CPP14_FLAG - Success -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Found Blitz: /home/mmm/devel/.conda/envs/xtensor-benchmark/include -- Found Armadillo: /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so (found version "9.200.7") -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2") -- Found PythonInterp: /home/mmm/.conda/envs/xtensor-benchmark/bin/python (found version "3.7.1") -- Found NumPy: version "1.18.5" /home/mmm/.conda/envs/xtensor-benchmark/lib/python3.7/site-packages/numpy/core/include CMake Error at modules/FindPythran.cmake:43 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:44 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:45 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:46 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:49 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:52 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:56 (string): string sub-command REGEX, mode MATCH regex "[0-9]*" matched an empty string. Call Stack (most recent call first): CMakeLists.txt:138 (find_package) CMake Error at modules/FindPythran.cmake:57 (math): math cannot parse the expression: "(NOTFOUND * 10000) + (NOTFOUND * 100) + NOTFOUND": syntax error, unexpected exp_TIMES (11). Call Stack (most recent call first): CMakeLists.txt:138 (find_package) COMPILING WITH ====================================== COMPILER : /usr/bin/c++ FLAGS : -Ofast -ffast-math -march=native -pthread Found xtensor : /home/mmm/.conda/envs/xtensor-benchmark/include Found xsimd : /home/mmm/.conda/envs/xtensor-benchmark/include Found eigen : /home/mmm/.conda/envs/xtensor-benchmark/include/eigen3 Found Blitz : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libblitz.so Found Armadillo : /home/mmm/devel/.conda/envs/xtensor-benchmark/include | /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so Found Pythran : Using benchmark : | benchmark::benchmark_main -- Configuring incomplete, errors occurred! See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeOutput.log". See also "/home/mmm/devel/xtensor-benchmark/CMakeFiles/CMakeError.log".

It seems like you can pass a -DPYTHON_LIBRARIES=path_to_python_lib option (or DPYTHON_LIBRARY for very olf version of cmake, i.e. < 3) to cmake. Let's hope it fixes the problem for good!

Unfortunately, no. We're able to set the correct pythonlibs path, but it is still somehow overriden by python 3.8?!

-- Found Blitz: /home/mmm/devel/.conda/envs/xtensor-benchmark/include -- Found Armadillo: /home/mmm/devel/.conda/envs/xtensor-benchmark/lib/libarmadillo.so (found version "9.200.7") -- Found PythonLibs: /home/mmm/.conda/envs/xtensor-benchmark/lib/python3.7 (found version "3.8.2") -- Found PythonInterp: /home/mmm/.conda/envs/xtensor-benchmark/bin/python (found version "3.7.1") -- Found NumPy: version "1.18.5" /home/mmm/.conda/envs/xtensor-benchmark/lib/python3.7/site-packages/numpy/core/include CMake Error at modules/FindPythran.cmake:43 (string): string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command. Call Stack (most recent call first): CMakeLists.txt:138 (find_package)

I tried to go the other way and install py3.8, but conda env has some unresolvable dependency conflicts and is unable to do that.

breznak added 4 commits June 4, 2020 23:27

google-benchmark from conda

1ab19c4

Blitz: add to conda env

2ea9c94

Pythran: install from conda

b9e36b9

not from pip

CMake: add conda's include,lib paths

253142e

breznak mentioned this pull request Jun 5, 2020

when do cmake,it will accur following error results #13

Open

JohanMabille requested changes Jun 5, 2020

View reviewed changes

environment.yml Outdated Show resolved Hide resolved

CMakeLists.txt Outdated Show resolved Hide resolved

CMakeLists.txt Outdated Show resolved Hide resolved

breznak added 3 commits June 5, 2020 10:06

cmake: fix Numpy detection

869ca2d

typo in package name, thanks Johan!

revert conda paths

f7fc77a

Revert "google-benchmark from conda"

3cb72bc

This reverts commit 1ab19c4.

breznak commented Jun 5, 2020

View reviewed changes

JohanMabille approved these changes Jun 11, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix dependencies - get from conda #14

Fix dependencies - get from conda #14

breznak commented Jun 5, 2020

JohanMabille left a comment

breznak left a comment

breznak Jun 5, 2020

breznak Jun 5, 2020

JohanMabille Jun 11, 2020

breznak Jun 12, 2020 •

edited

Loading

JohanMabille Jun 12, 2020 •

edited

Loading

breznak Jun 12, 2020

JohanMabille Jun 23, 2020 •

edited

Loading

breznak Jun 23, 2020

JohanMabille Jun 23, 2020

breznak Jun 23, 2020

		@@ -135,8 +134,8 @@ endif()

		if(BENCHMARK_PYTHONIC)
		find_package(PythonLibs)

Fix dependencies - get from conda #14

Are you sure you want to change the base?

Fix dependencies - get from conda #14

Conversation

breznak commented Jun 5, 2020

JohanMabille left a comment

Choose a reason for hiding this comment

breznak left a comment

Choose a reason for hiding this comment

breznak Jun 5, 2020

Choose a reason for hiding this comment

breznak Jun 5, 2020

Choose a reason for hiding this comment

JohanMabille Jun 11, 2020

Choose a reason for hiding this comment

breznak Jun 12, 2020 • edited Loading

Choose a reason for hiding this comment

JohanMabille Jun 12, 2020 • edited Loading

Choose a reason for hiding this comment

breznak Jun 12, 2020

Choose a reason for hiding this comment

JohanMabille Jun 23, 2020 • edited Loading

Choose a reason for hiding this comment

breznak Jun 23, 2020

Choose a reason for hiding this comment

JohanMabille Jun 23, 2020

Choose a reason for hiding this comment

breznak Jun 23, 2020

Choose a reason for hiding this comment

breznak Jun 12, 2020 •

edited

Loading

JohanMabille Jun 12, 2020 •

edited

Loading

JohanMabille Jun 23, 2020 •

edited

Loading