Python Version and General Setup Help

[mandos21]

Hey, I'm looking to get started with this project to integrate pronoun coreferencing via another pythonm library, neuralcoref, into an existing c project. I'm using python 3.7.16, and I was wondering which version you have tested this with, and if this version is supported. I'm running into the following errors, and I'm not sure if they're due to my environment, configuration or user error. I don't have a lot of C/++ experience, so apologies in advance if I'm missing something obvious here. Thanks!

/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_start_new_thread':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:203: undefined reference to 'pthread_attr_setstacksize'
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:222: undefined reference to 'pthread_create'
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:239: undefined reference to 'pthread_detach'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_allocate_lock':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:292: undefined reference to 'sem_init'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_free_lock':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:317: undefined reference to 'sem_destroy'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_acquire_lock_timed':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:366: undefined reference to 'sem_timedwait'
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:369: undefined reference to 'sem_trywait'
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:372: undefined reference to 'sem_wait'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_release_lock':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:441: undefined reference to 'sem_post'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_create_key':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:677: undefined reference to 'pthread_key_create'
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:682: undefined reference to 'pthread_key_delete'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_set_key_value':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:712: undefined reference to 'pthread_setspecific'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_tss_create':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:750: undefined reference to 'pthread_key_create'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_tss_delete':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:767: undefined reference to 'pthread_key_delete'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_tss_set':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:776: undefined reference to 'pthread_setspecific'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function '_pythread_pthread_set_stacksize':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:640: undefined reference to 'pthread_attr_setstacksize'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_delete_key':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:696: undefined reference to 'pthread_key_delete'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_delete_key_value':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:704: undefined reference to 'pthread_setspecific'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_get_key_value':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:723: undefined reference to 'pthread_getspecific'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(thread.o): In function 'PyThread_tss_get':
/home/admin/mattd/Python-3.7.16/Python/thread_pthread.h:784: undefined reference to 'pthread_getspecific'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(dynload_shlib.o): In function '_PyImport_FindSharedFuncptr':
/home/admin/mattd/Python-3.7.16/./Python/dynload_shlib.c:83: undefined reference to 'dlsym'
/home/admin/mattd/Python-3.7.16/./Python/dynload_shlib.c:96: undefined reference to 'dlopen'
/home/admin/mattd/Python-3.7.16/./Python/dynload_shlib.c:127: undefined reference to 'dlsym'
/home/admin/mattd/Python-3.7.16/./Python/dynload_shlib.c:96: undefined reference to 'dlopen'
/home/admin/mattd/Python-3.7.16/./Python/dynload_shlib.c:102: undefined reference to 'dlerror'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(posixmodule.o): In function 'os_openpty_impl':
/home/admin/mattd/Python-3.7.16/./Modules/posixmodule.c:5954: undefined reference to 'openpty'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(posixmodule.o): In function 'os_forkpty_impl':
/home/admin/mattd/Python-3.7.16/./Modules/posixmodule.c:6048: undefined reference to 'forkpty'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(signalmodule.o): In function 'signal_pthread_sigmask_impl':
/home/admin/mattd/Python-3.7.16/./Modules/signalmodule.c:855: undefined reference to 'pthread_sigmask'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(signalmodule.o): In function 'signal_pthread_kill_impl':
/home/admin/mattd/Python-3.7.16/./Modules/signalmodule.c:1124: undefined reference to 'pthread_kill'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(timemodule.o): In function 'time_pthread_getcpuclockid':
/home/admin/mattd/Python-3.7.16/./Modules/timemodule.c:342: undefined reference to 'pthread_getcpuclockid'
/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(faulthandler.o): In function 'faulthandler_thread':
/home/admin/mattd/Python-3.7.16/./Modules/faulthandler.c:564: undefined reference to 'pthread_sigmask'
collect2: error: ld returned 1 exit status
gmake[3]: *** [coreferencer_cpp_exec] Error 1
gmake[2]: *** [CMakeFiles/coreferencer_cpp_exec.dir/all] Error 2
gmake[1]: *** [CMakeFiles/coreferencer_cpp_exec.dir/rule] Error 2
gmake: *** [coreferencer_cpp_exec] Error 2

Here is my main.cpp:

#include <iostream>
#define SPACY_HEADER_ONLY
#include "src/spacy/spacy"

void test() {
    Spacy::Spacy spacy;
    auto nlp = spacy.load("en_core_web_lg");
    auto doc = nlp.parse("This is a sentence.");
    for (auto& token : doc.tokens())
        std::cout << token.text() << " [" << token.pos_() << "]\n";
}

int main() {
    test();
}

And my CMakeLists.txt

cmake_minimum_required(VERSION 3.21)
project(coreferencer_cpp_exec)

set(CMAKE_CXX_STANDARD 14)

add_executable(coreferencer_cpp_exec main.cpp)

include_directories(/usr/local/include/python3.7m/)
include_directories(/usr/local/lib/python3.7/site-packages)

target_link_libraries(coreferencer_cpp_exec LINK_PUBLIC python3.7m -L/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu)
target_link_libraries(coreferencer_cpp_exec LINK_PUBLIC python3.7m -L/usr/local/lib)

[mandos21]

I solved it (partially). In case anyone happens to stumble across this, adding the following lines to my CMakelists file resolved the errors, but now it is using the wrong python version.

target_link_libraries(coreferencer_cpp_exec LINK_PUBLIC dl)
target_link_libraries(coreferencer_cpp_exec LINK_PUBLIC util)
target_link_libraries(coreferencer_cpp_exec LINK_PUBLIC pthread)
target_link_libraries(coreferencer_cpp_exec LINK_PUBLIC m)
target_link_libraries(coreferencer_cpp_exec LINK_PUBLIC rt)

[d99kris]

Hi @mandos21 - glad to hear you found a solution. Let me look into updating the README.md for the case of using SPACY_HEADER_ONLY to cover the required linking.

As for your remaining problem

now it is using the wrong python version

feel free to share more details and perhaps I, or someone, can help.

Edit: For the Python version issue, it could be useful to know which Linux distro is used and how Python was installed. Nowadays most distros has moved away from Python2, so there shouldn't generally be much issues. In your logs I see both paths to a system installed Python3 (/usr/local/lib/python3.7) and a one under the home (/home/admin/mattd/Python-3.7.16) directory, so that could cause problems. It's generally easier if just sticking to the Python3 provided by the OS distro.

[mandos21]

Hi @d99kris, thanks for getting back. I am on centos 7, Kernel Linux 3.10.0-1127.el7.x86_64. Interesting catch with that second path to python in the home directory. I'm not sure why it was trying to use that, that's where I built python from to install on this machine. But I deleted those directories and haven't seen any changes in the output yet. Right now I'm up against this error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  No module named spacy. Try: pip3 install -U spacy

There are three versions of python installed, and I don't have the option of removing the other installs to simplify things.

$ python
Python 2.7.5 (default, Apr  2 2020, 13:16:51)

$ python3
Python 3.7.16 (default, Jan 24 2023, 22:07:23)

$ python3.6
Python 3.6.8 (default, Nov 16 2020, 16:55:22)

//calling from the given path reveals why the program is calling the wrong python instance

$ /usr/local/bin/python3
Python 3.7.16 (default, Jan 24 2023, 22:07:23)

$ /usr/bin/python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22)

Spacy, is installed for python3.7 only. I would use 3.6, but neuralcoref only cooperates on 3.7:

$ python -m spacy
/usr/bin/python: No module named spacy

$ python3.6 -m spacy
/usr/bin/python3.6: No module named spacy

$ python3 -m spacy

ℹ Available commands
download, link, info, train, pretrain, debug-data, ud-train, evaluate,
ud-evaluate, convert, package, init-model, profile, validate

It looks like the program is using the /usr/bin version of python3, not the /usr/local/bin version, which I imagine is expected behavior. Is there any way of specifying which python installation spacy-cpp uses?

[d99kris]

Is there any way of specifying which python installation spacy-cpp uses?

I think it's simply controlled by the include/lib path order when it comes to libpython. So it should be possible to specify by putting the preferred libpython paths first in include/lib paths while building. I don't know how to achieve this in CMake when one has multiple Python system installations. For a standalone compilation on the command line directly using clang++/g++ it should be fairly straightforward though, using the -I and -L arguments.

Btw, in case it helps; one can check which python has been linked in by running ldd on an executable (or library).

[mandos21]

Here is my output for ldd on my application, no mention of python for some reason.

$ ldd coreferencer_cpp_exec
	linux-vdso.so.1 =>  (0x00007fff379d6000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f98ebc37000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f98eba34000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f98eb818000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f98eb610000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f98eb308000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f98eb006000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f98eadf0000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f98eaa22000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f98ebe3b000)

Forgive my ignorance if I'm not doing this right, I'm still very new to c++/libraries/compilation, most of my experience is with Ruby and Python. Using the CLion debugger I was able to verify the version of python that is being accessed via PY_MAJOR, MINOR, and MICRO _Version, and it is 3.7.16, the version that has spacy installed.

[mandos21]

A couple more things I've tried that haven't worked:

Recompiled spacy-cpp with exact version numbers in the makefile:

find_package(PythonLibs 3.7.16 REQUIRED)
find_package(PythonInterp 3.7.16 REQUIRED)

Linking the directories directly:

target_link_libraries(spacy python3.7m -L/usr/local/lib/python3.7/config-3.7m-x86_64-linux-gnu)
target_link_libraries(spacy python3.7m -L/usr/local/lib)

Changing the symlinks in /usr/bin and /bin that pointed python3 to python3.6 to python3.7:

[/usr/bin]$ ls -la python*
lrwxrwxrwx. 1 root root        7 Nov 10 13:32 python -> python2
lrwxrwxrwx. 1 root root        9 Nov 10 13:32 python2 -> python2.7
-rwxr-xr-x. 1 root root     7144 Apr  2  2020 python2.7
lrwxrwxrwx  1 root root        9 Jan 27 21:12 python3 -> python3.7
-rwxr-xr-x. 2 root root    11328 Nov 16  2020 python3.6
lrwxrwxrwx  1 root root       17 Nov 10 15:30 python3.6-config -> python3.6m-config
-rwxr-xr-x. 2 root root    11328 Nov 16  2020 python3.6m
-rwxr-xr-x  1 root root      173 Nov 16  2020 python3.6m-config
-rwxr-xr-x  1 root root     3403 Nov 16  2020 python3.6m-x86_64-config
-rwxr-xr-x  1 root root 13325864 Jan 27 21:09 python3.7
-rwxr-xr-x  1 root root     2911 Jan 27 21:09 python3.7-config
-rwxr-xr-x  1 root root 13325864 Jan 27 21:09 python3.7m
-rwxr-xr-x  1 root root     2911 Jan 27 21:09 python3.7m-config
lrwxrwxrwx  1 root root       16 Jan 27 21:13 python3-config -> python3.7-config

So far nothing. Why can't python find spacy?

[d99kris]

Why can't python find spacy?

Python will not be able to find spacy if spacy was installed using a different python version AFAIK.

Recompiled spacy-cpp with exact version numbers in the makefile:

Note that compiled spacy-cpp is only used if you don't define SPACY_HEADER_ONLY. Either one use a compiled version of spacy-cpp or otherwise a header-only. Assuming the spacy-cpp CMakeLists.txt was edited, it will only affect you when using spacy-cpp as a "real" library, i.e. not defining SPACY_HEADER_ONLY. Hope this helps.

[d99kris]

Hi @mandos21 - just wanted to mention that I've added two simple CMake examples for using spacy-cpp:
Using spacy-cpp as header-only library
Using spacy-cpp as shared library

They provide bare-bones CMakeLists.txt for using spacy-cpp. If starting a new project I recommend using spacy-cpp as a shared library (rather than header-only).

Not sure if you're still trying to get it to work, but it might help someone else encountering this thread.