Undefined references to some absl functions when building from scratch

[mdorier]

I'm trying to build redpanda from scratch. I have the dependencies installed (cmake doesn't complain about any of them missing), but when I do "make", at about 30% I end up with this:

[ 31%] Linking CXX executable test_secure_hashing_rpunit
ld.lld: error: undefined symbol: absl::lts_20240722::container_internal::kSooControl
>>> referenced by crypto.cc
>>>               crypto.cc.o:(crypto::internal::get_md(crypto::digest_type)) in archive ../../crypto/libv_v_crypto.a

And a bunch of other undefined references, all related to absl. I have abseil 20240722.0 installed (built as static libraries), and cmake found it when I called cmake. Any idea why this happens?

[mdorier]

Ok I figured it out: I install abseil 20240722.0 as a dependency to something (protocol buffer I think), so it was available for find_package to find. But redpanda's dependencies.cmake downloads its own abseil (20230802.1) as part of the build process, and I guess it then includes the external abseil but tries to link against the internal one. I fixed it by making redpanda download the same version I had.

Now I'm getting another undefined reference:

[ 35%] Linking CXX executable zstd_tests_rpunit
ld.lld: error: undefined symbol: typeinfo for snappy::Sink
>>> referenced by snappy_standard_compressor.cc
>>>               snappy_standard_compressor.cc.o:(typeinfo for compression::snappy_iobuf_sink) in archive ../libv_v_compression.a
>>> did you mean: vtable for snappy::Sink
>>> defined in: /home/ubuntu/Code/inst-redpanda/spack/opt/spack/linux-ubuntu22.04-aarch64/gcc-13.1.0/snappy-1.2.1-2pnuhyd4nf3vvike7zxyzkebtd5cjwog/lib/libsnappy.so.1.2.1

ld.lld: error: undefined symbol: typeinfo for snappy::Source
>>> referenced by snappy_standard_compressor.cc
>>>               snappy_standard_compressor.cc.o:(typeinfo for compression::snappy_iobuf_source) in archive ../libv_v_compression.a
>>> did you mean: vtable for snappy::Source
>>> defined in: /home/ubuntu/Code/inst-redpanda/spack/opt/spack/linux-ubuntu22.04-aarch64/gcc-13.1.0/snappy-1.2.1-2pnuhyd4nf3vvike7zxyzkebtd5cjwog/lib/libsnappy.so.1.2.1
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

As far as I know, redpanda isn't building snappy by itself, I installed it myself. I tried building it as a shared library and as a static one, the problem is the same. Undefined typeinfo is usually the sign of having compiled with -fno-rtti, but I it doesn't seem that snappy uses -fno-rtti unless it's on Windows.

[mdorier]

I didn't read snappy's cmake file right, it disables RTTI on all platforms. I had to manually edit its cmake file and build it manually. Now on to the next problems...

[rockwotj]

We are updating our build to using Bazel - can you give that a try?

[mdorier]

I started giving it a try but for now I have some errors from Bazel (and it's not as easy to understand where they are coming from as when using cmake). I'm looking at the install-dependencies.sh file to try to understand what I should install. I am on a supercomputer without admin access so I can't use that file to install them, I have to build them myself, including their dependencies, using the Spack package manager. If you happen to have a list of these dependencies (rather than a list of packages for specific Linux distributions), could you point me to it?

[rockwotj]

Sorry I don't have a list of dependencies needed. I might suggest if you have access to docker your try running in an ubuntu container, otherwise that is not a build environment we use nor have any knowledge of! If you run into specific build errors I am happy to help diagnose however.

[dotnwat]

@mdorier the list of dependencies are in bazel/install-deps.sh. as far as helping goes, like @rockwotj mentioned, it may be useful to do this in a docker container and then you can share the dockerfile with us so that we can reproduce the issues locally.