World state of build systems and our context

[prasannavl]

Summary

• Creating this as living thread to document some of shenanigans of cross builds along with multiple languages.
• This is key place to document build system and cross toolchain related expectations, tips, drill downs and relevant issues
• Note: WIP

Why

• The dependency chain and cross language toolchains across different arch are complex to integrate well.
• The is made worse, by lack of good high-level documentation across the cross compilation nuances in toolchains, often have to dig into the code of LLVM or clang is required to get the accurate behaviour of a particular version.
• The fragile chaining of many make files (currently, we have culled down and cleaned up path dependencies significantly) and the complex autotools based builds in theory work, but minor changes trip up things across different places, making it expensive to fix in a way that's maintainable long term, esp with compilation expectations from integrating multiple lang ecosystems that go beyond the usual paradigms of autotools (eg: autotools -> cargo -> build_script -> autotools).
• Modern build systems like bazel and buck2 are built to handle this, but before we plan a leap, clarifying the current context is needed to maintain.

World State

C++

Usual dependencies (Taking libc out of the equation):

• GCC
  • libstdc++
  • libgcc_s // Compiler built-ins + unwind support
  • ...
• Clang (Without using gcc -- if using the GCC driver, same as above)
  • Compiler built-ins
    • This is provided for each arch in the clang source, but most dist only provide built versions for the host, needs to be rebuilt for each target for cross compilation, which results in a large chain of toolchain rebuilds
  • libunwind
    • LLVM will need this first to add the correct stack unwinding behaviour for the other core dependencies
  • libc++abi
  • libc++
  • compiler-rt

Related

• https://github.com/uber/hermetic_cc_toolchain (Zig toolchain bazel rules)

Useful ways to drill down

• clang++ -### ./main.cpp -fuse-ld=lld
• clang++ -### ./main.cpp --target=aarch64-unknown-linux-gnu -fuse-ld=lld to observe target differences.
• g++ -### ./main.cpp
• clang++ ./main.cpp -fuse-ld=lld --print-search-dirs -v

Rust

• Rust toolchain which internally uses LLVM 15 currently (uses it in the lib form), currently works well with gcc toolchains, however I still have to dig in on the state of things without gcc.

Dependencies

• libdl:
  • Used for getting backtraces on panic (dl_iterate_phdr)
• libpthread:
  • Used for sync of backtraces
• librt:
  • clock_gettime
• libgcc_s:
  • Unwind, etc.

Related

• Refactor the linkage handling of libunwind rust-lang/rust#84040
• 1.46.0 fails to build with llvm-libunwind enabled rust-lang/rust#76020
• "Hello world!" binary links a bunch of unnecessary libraries rust-lang/rust#29482
• Provide a way of linking libgcc statically rust-lang/rust#29527
• https://users.rust-lang.org/t/rustc-statically-links-libdl-a-rather-than-dynamic-link-libdl-sl/22154
• https://bugs.gentoo.org/733242

Our build system

TODO

Unresolved questions

• Should we move to bazel or buck2?
  • Bazel has rust_rules which uses cargo_universe to generate rules for cargo (preceded by cargo_raze as Google's experiment)
  • buck2 way is to use reindeer to generate rules for cargo packages
  • Both Google and Facebook have significant Rust investments that these can be relied on.
• Should we move to meson?
  • The beloved alternative in the non mono repo OSS world is meson (systemd, gnome, etc..), which is also a good option, however doesn't do well with cargo and there has been a long on-going conversations on how to support this.
  • The community doesn't appear thrilled on the cargo approach and instead of cargo_universe, and reindeer style approach, has rather sadly debating rust community's friendliness and being dogmatic on it's view of how dependencies needs to be built which has both it's pros and cons.
  • Eg: Rust: Support external crates mesonbuild/meson#2173
• Is just using cargo to drive the compilation with build.rs, and building C++ a practical approach? It is far more maintainable and robust than autotools and most of the *-sys crates are built with it after all.
  • However, we'll have to implement our own dep resolution for incremental build support on top of cc-rs, which has both it's pros and cons - can benefit other C++/Rust interop projects.
• Should we use the zig cc toolchain to drive the C++ compilation?