ImportError: /lib/x86_64-linux-gnu/liblapacke.so.3: undefined symbol: cgesvdq_

[sheyu921]

Hi:
I can not import the pygsvd, the errors show as blew:

ImportError Traceback (most recent call last)
in
----> 1 import pygsvd as gsvd

~/miniconda3/lib/python3.8/site-packages/pygsvd.py in
1 import numpy as np
----> 2 import _gsvd
3
4 def gsvd(A, B, full_matrices=False, extras='uv'):
5 '''Compute the generalized singular value decomposition of

ImportError: /lib/x86_64-linux-gnu/liblapacke.so.3: undefined symbol: cgesvdq_

What should I do? Thank you!

[bnaecker]

Hi @sheyu921! Sorry to hear you're having trouble.

Could you please provide details of:

• The platform you're using
• How you built the package
• The output of running the ldd command on the compiled shared library that this project builds. File should be named something like _gsvd.cpython-38m-x86_64-linux-gnu.so, and it'll be in the path where you built the package.

[sheyu921]

Thanks for your response so quickly, and I am sorry not to supply the details.

1. The platform I am using is Ubuntu20.04 (wsl).

2. I built the package by using the method you recommend.
   First, I install the liblapacke by "sudo apt install liblapacke-dev",
   Second, I used "dpkg -L liblapacke" to find the location of the lib showing as below

(base) syw@dell-office:/mnt/c/Users/sheya/Desktop/pygsvd-master$ dpkg -L liblapacke
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/liblapacke.so.3.9.0
/usr/share
/usr/share/doc
/usr/share/doc/liblapacke
/usr/share/doc/liblapacke/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/liblapacke
/usr/lib/x86_64-linux-gnu/liblapacke.so.3
/usr/share/doc/liblapacke/changelog.Debian.gz

Finally, I change the content of "setup.cfg" as below:

[build_ext]
define=USE_LAPACK
include_dirs=/usr/include
library_dirs=/usr/lib/x86_64-linux-gnu

Then, using the python3 setup.py build_ext and python3 setup.py {install,develop} to build the package.

I used the 'ldd _gsvd.cpython-38-x86_64-linux-gnu.so', and the results show as bellow:

(base) syw@dell-office:/mnt/c/Users/sheya/Desktop/pygsvd-master/build/lib.linux-x86_64-3.8$ ldd _gsvd.cpython-38-x86_64-
linux-gnu.so
        linux-vdso.so.1 (0x00007ffcb45e8000)
        liblapacke.so.3 => /lib/x86_64-linux-gnu/liblapacke.so.3 (0x00007fcc13a3d000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcc13a1a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcc13828000)
        libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007fcc11e29000)
        liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007fcc11785000)
        libtmglib.so.3 => /lib/x86_64-linux-gnu/libtmglib.so.3 (0x00007fcc1171f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcc13ca3000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcc115d0000)
        libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007fcc11308000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fcc112ed000)
        libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007fcc112a3000)

Looks like the shared libs have been linked correctly. However, when I import pygsvd, the errors show as below:

Python 3.8.5 (default, Sep  4 2020, 07:30:14)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygsvd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/c/Users/sheya/Desktop/pygsvd-master/pygsvd.py", line 2, in <module>
    import _gsvd
ImportError: /lib/x86_64-linux-gnu/liblapacke.so.3: undefined symbol: cgesvdq_

What should I do? Changing the version of python will solve the problem? Thank you!.

[bnaecker]

Thanks for the detailed information, that's very helpful. I'm not sure if using WSL changes the story at all, but we can try to debug assuming it does not, acting as if you're working on a vanilla Ubuntu distribution.

The underlying issue that your version of LAPACK build of the extension module contains an undefined symbol: cgesvdq. However, the extension module provided by this package doesn't explicitly use that function. To see that, if you call nm _gsvd.cpython-38-x86_65-linux-gnue.so, you should see only two symbols from the LAPACK library at all: zggsvd3 and dggsvd3. These are the complex and real versions of the underlying generalized SVD routine that this library wraps. The missing symbol is for a particular type of singular value decomposition, and is unrelated to this package. Please include the output of the nm call above here, just to make sure I'm not missing something about how the extension itself is built on your platform.

It would be helpful to know if you can compile and link a small example executable successfully against your LAPACK implementation at all. You should be able to build and run the following example without any compilation or linking errors. The example will segfault immediately, so don't worry about that. We're just testing if we can successfully use LAPACK on your system, independent of the extension module.

#include "lapacke.h"

int main(int argc, char** argv) {
	int ret = LAPACKE_dggsvd3(
		LAPACK_ROW_MAJOR, 'N', 'N', 'N', 0, 0, 0, NULL, NULL, NULL, 0, NULL, 0, NULL,
		NULL, NULL, 0, NULL, 0, NULL, 0, NULL
	);
}

Compile and run with gcc -o example example.c -llapacke && ./example, and let me know what happens! This will help determine if the issue is the extension itself, or something I don't understand about linking against LAPACK on your platform.

[3togo]

I am using ubuntu 22.04

eli@eli-A17-R4:~/temp$  gcc -o example example.c -llapacke
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/liblapacke.so: undefined reference to `dgetsqrhrt_'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/liblapacke.so: undefined reference to `cungtsqr_row_'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/liblapacke.so: undefined reference to `zgetsqrhrt_'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/liblapacke.so: undefined reference to `dorgtsqr_row_'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/liblapacke.so: undefined reference to `sorgtsqr_row_'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/liblapacke.so: undefined reference to `cgetsqrhrt_'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/liblapacke.so: undefined reference to `zungtsqr_row_'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/liblapacke.so: undefined reference to `sgetsqrhrt_'
collect2: error: ld returned 1 exit status

[bnaecker]

@3togo Sorry to hear you're having issues. I've not looked at this issue in quite some time now, but I'll try to assist.

You're seeing a failure to link the executable. The executable depends on liblapacke.so, which itself has references to those symbols we're seeing, but which are defined in other shared objects. You will need to figure out which libraries liblapacke.so expects, usually by running ldd against it, and then figure out where those are installed on your machine. With that in hand, you can run gcc -o example example.c -L path/to/lapack/libs -llapacke. That should allow gcc to find those libraries and link them appropriately. You may need to specify more than one directory with -L.

To expand a bit, liblapacke.so (note the trailing "e") is the C bindings to the actual Fortran implementation of LAPACK. The latter is usually distributed at the same time, in a library usually named liblapack.so. It should live right next to the C bindings -- that is, you should likely see both liblapacke.so and liblapack.so in /usr/lib/x86_64-linux-gnu/. If that's true, then adding -L/usr/lib/x86_64-linux-gnu to the gcc invocation should be enough for you to compile and link the test example I posted above.

Also, I'm not entirely clear why this should be needed. If you installed lapacke via apt, then the library should be on the compiler's search path by default, and you should not need to specify a library path with -L. I've just tested this on an Ubuntu 22.04 VM, and gcc -o example example.c -llapacke is sufficient to build and link successfully. Did you happen to install liblapacke in a different way than apt? Did you build / install LAPACK from source?

While I'm here, @sheyu921 did you ever get this issue resolved?

[3togo]

ubuntu 22.10 might be very different.

gcc -o example example.c -llapacke => won't work
gcc -o example example.c -llapack => could compile but with segmentation fault

0x00007fffee7a0193 in mkl_lapack.dggsvd3_ () from /lib/x86_64-linux-gnu/libmkl_intel_lp64.so
(gdb) bt
#0 0x00007fffee7a0193 in mkl_lapack.dggsvd3_ () from /lib/x86_64-linux-gnu/libmkl_intel_lp64.so
#1 0x00007ffff79cc4c2 in mkl_lapack.dggsvd3_ () from /lib/x86_64-linux-gnu/libmkl_rt.so
#2 0x00007fffee945e86 in LAPACKE_dggsvd3_work () from /lib/x86_64-linux-gnu/libmkl_intel_lp64.so
#3 0x00007fffee945576 in LAPACKE_dggsvd3 () from /lib/x86_64-linux-gnu/libmkl_intel_lp64.so
#4 0x00005555555551a1 in main ()
(gdb)
[1]+ Stopped

[bnaecker]

@3togo It would help a great deal if you could provide answers to my questions above. How did you install LAPACK itself on your system? Through apt or from sources? Based on your last message, it appears you're using Intel's MKL implementation of LAPACK. That should be OK, but I've done less testing with it.

gcc -o example example.c -llapacke => won't work

What does "won't work" mean here, exactly?

gcc -o example example.c -llapack => could compile but with segmentation fault

This segfault should be very easy to track down. Can you locate the invalid pointer that's being dereferenced? That should be some pointer that example.c doesn't correctly allocate. Also, to be clear, that example is not very relevant. I put it together to test basic compilation and linking failures. Does the actual extension library we care about here build and run correctly?