build: Allow simple fixed library names (as used in pkgsrc) #4252
Conversation
This is the current state of OpenBLAS library naming for the pkgsrc packages (NetBSD and cross-platform). The idea is to be able to 1) be able to have a stable library name to expect from the build 2) install multiple variants next to each other (single-threaded, parallel) For this, the new switch FIXED_LIBNAME is introduced. The build uses make FIXED_LIBNAME=1 LIBNAMESUFFIX=openmp In the case of 64 bit indices, it uses make FIXED_LIBNAME=1 LIBNAMESUFFIX=openmp INTERFACE64=1 LIBSOBASENAME=openblas64 This results in what I assumed to be the intention of the name variables to begin with, namely primary predictable library names like libopenblas_openmp.so libopenblas64_openmp.so (As pkgsrc keeps a list of expected file names resulting from a build on the user machine, too funky names with CPU model in there and symlinks are too funky.) I see that the naming scheme of the 64 bit indices lib is still under discussion, along with symbol suffix. So far, we use the names without renamed symbols. They are all installable at the same time and user software can choose which one to use, also using the pkg-config file (which could be handled more elegantly). Without an established convention, I didn't think like we should make the call. Depends on what downstream packages use at some point. I don't say that I do not cause more mess than necessary, or that I fully understand the already pesent mess. Just throwing this suggestion out there. Maybe my the FIXED_LIBNAME switch or something along that can be included. Or we work out something better that fulfills the demands of a pkgsrc installation. See the github mirror of pgksrc for the patches in use. https://github.com/NetBSD/pkgsrc/tree/trunk/math/openblas https://github.com/NetBSD/pkgsrc/tree/trunk/math/openblas64 https://github.com/NetBSD/pkgsrc/tree/trunk/math/openblas_openmp https://github.com/NetBSD/pkgsrc/tree/trunk/math/openblas64_openmp https://github.com/NetBSD/pkgsrc/tree/trunk/math/openblas_pthread https://github.com/NetBSD/pkgsrc/tree/trunk/math/openblas64_pthread Also, I hope I didn't break any other configuration. And, lastly, one idea: If we settle on OpenMP always and the ILP64 symbols with suffix, we could just get away with building _one_ libopenmp.so (with libtool versioning, perhaps, libopenblas.so.0.4.33 and symlinks …). That would also be great and lots of Makefile machinery could go. Until then, please consider my hack on top;-)
|
Thanks for sharing the pkgsrc needs and details @drhpc, interesting. I tried this PR with the given instructions above - the build works but the install doesn't; the static library name isn't yet handled correctly it looks like:
This should be
You can already achieve this by just renaming the library after it's built, right? From my point of view, having to support a lot of ad-hoc naming conventions that different distros already make (I need to support auto-detection and linking against OpenBLAS in NumPy, SciPy and Meson), a free-form The current convention is
That seems very reasonable. I think any fixed naming works. MKL has a very structured scheme that I think is clear; doing something similar but using
OpenMP isn't always an option - for example on PyPI, where it's not possible to distribute an OpenMP runtime as a separate package, and vendoring it is not pretty (so we prefer pthreads instead for PyPI). |
|
Actually I'd prefer that distributions rename or softlink the current, limited set of library names according to their needs, rather than add another option. And I assume the current situation must be bad enough for third parties like flexiblas |
|
@rgommers About static libs: You missed giving the setttings also to the install make process. This is transient variables — that is why you could write it into Makefile.rule to persist. This obviously misses |
|
Compiler-specific variants next to each other: Oh, wow. I can even be more complex! @rgommers are you referring only to the OpenMP runtime itself or also having OpenBLAS built with a differing compiler? In any case, I think that's too far. I don't see other projects branching out installs for differing OpenMP runtimes. It is already a rare nuisance that a library has serial and parallel variants (right now, I can only think of fftw as another example, with similar naming issues … threads, MPI, single, double quad precision). I'd rather have that space reduced than enlarged. For a packager, it is weird to re-build a library for a differing API/ABI subset and have use cases for both variants in the same system. I know where it comes from. My own atmospheric model code of course knows a switch to build with single or double precision. That's what scientists play with. But once such code enters mainstream system installs, it's weird. Offering the same API and ABI for differing data types and other conditions is a nightmare for keeping track of who is crashing because of which code. That's why a symbol suffix is discussed/supported for ILP64 but not a standard from the beginning, sadly. With the suffix there, one could easily roll the 64 bit interface into the same library binary and just offer libopenblas without confusion. One could imagine adding openmp and pthread API into the same binary, for the somewhat ugly sideeffect that a serial or pthread application now links with openmp runtime for the shared lib. (PyPI cannot ship an OpenMP runtime!? I did not think about that angle before. I must admit that openmp runtime as separate thing only entered my mind with pkgsrc, which caters for NetBSD installs that indeed miss libgomp. My world starts with a compiler toolchain, MPI library on top … and naturally includes an OpenMP lib. But of course, the world is wonderfully messy and colourful.) Now, if there is a standard switch for non-conflicting library names and I can predict, I'd use it. It will be a annoying to switch to names that differ from what we settled on, but most pkgsrc installs use plain netlib where the user doesn't care, and people dealing with scientific code are used to some pain. @martin-frbg The problem I had with the current names is that they are not (easily) predictable. If building with I can derive the If there were a fixed scheme that
I could use it for a build in pkgsrc that wants to know the resulting file names in advance. I won't die on the hill about the exact naming scheme. I don't like introducing additional switches, but I saw all that complicated naming machinery and saw how I could bend it to my will. It would have been fine would it always just be libopenblas.so.0 for me to rename. But thinking about If all variants have SONAME So I add to my list above: The standard build please have an SONAME that matches the primary library name. Binary distros that offer one choice of BLAS at a time via alternatives symlinks already hack the builds to always provide SONAME=libblas.so, right? Ah, libblas.so.3 it is … to match what current netlib does. That number is a mess in itself, having to ensure that the ABI really matches. But this is not the way we choose in pkgsrc, and more detailed in my installs of it in HPC environments. I focus on a build environment where people can choose which toolchain to use and which library to link into their applications. Each user does that by choosing environments with environment modules, but also via specific BLAS library names. Providing a separate environment for all possible choices wouldn't scale. As long as not everybody is shipping their own binary container, these two approaches to installing BLAS should prevail. One wants to clearly tell variants apart, but offer them at the same time, the other wants to blur the distinction to be able to fling binary symlinks around on a per-user (single-user Linux system) install. |
|
superseded by my #4485 , I think, though my approach may be less elegant ? |
This is the current state of OpenBLAS library naming for the pkgsrc packages (NetBSD and cross-platform).
The idea is to be able to
For this, the new switch FIXED_LIBNAME is introduced. The build uses
make FIXED_LIBNAME=1 LIBNAMESUFFIX=openmp
In the case of 64 bit indices, it uses
make FIXED_LIBNAME=1 LIBNAMESUFFIX=openmp INTERFACE64=1 LIBSONAMEBASE=openblas64
This results in what I assumed to be the intention of the name variables to begin with, namely primary predictable library names like
libopenblas_openmp.so
libopenblas64_openmp.so
(As pkgsrc keeps a list of expected file names resulting from a build on the user machine, too funky names with CPU model in there and symlinks are too funky.)
I see that the naming scheme of the 64 bit indices lib is still under discussion, along with symbol suffix. So far, we use the names without renamed symbols. They are all installable at the same time and user software can choose which one to use, also using the pkg-config file (which could be handled more elegantly). Without an established convention, I didn't think like we should make the call. Depends on what downstream packages use at some point.
I don't say that I do not cause more mess than necessary, or that I fully understand the already pesent mess. Just throwing this suggestion out there. Maybe my the FIXED_LIBNAME switch or something along that can be included. Or we work out something better that fulfills the demands of a pkgsrc installation.
See the github mirror of pgksrc for the patches in use.
Also, I hope I didn't break any other configuration.
And, lastly, one idea: If we settle on OpenMP always and the ILP64 symbols with suffix, we could just get away with building one libopenmp.so (with libtool versioning, perhaps, libopenblas.so.0.4.33 and symlinks …). That would also be great and lots of Makefile machinery could go. Until then, please consider my hack on top;-)