Rust: Support external crates

[zeenix]

I was told that support of Rust was experimental so I can expect hickups and that's understandable but currently it seems external crates are not supported. Since you can't really do much in Rust without external crates, it basically means that Meson doesn't really support Rust. IMO until this issue is addressed in some way, support for Rust shouldn't be advertised at all.

[TingPing]

Correct me if I am wrong but Cargo simply just builds all dependencies and there is no concept of external dependencies (ala pkgconfig) right? Does Rust even have any standard directory to install shared libraries? If so we should support rust.find_library().

Otherwise just like Cargo to build all dependencies then all dependencies have to use Meson.

[tp-m]

I think what's needed is more something that does what Cargo does - downloads crates as per Cargo.toml and Cargo.lock and builds them / links them in as needed. This would be really powerful, esp. for mixed C/Rust projects.

[TingPing]

So we either:

• Re-implement Cargo
• Simply call Cargo making the Meson integration worthless

[zeenix]

@TingPing typically they just do static linking in Rust currently. cargo will simply download, build all deps and then take care of linking your binaries.

[zeenix]

@TingPing yeah but please keep in mind that this is useful for projects that will be a mix of Rust and C (or some other language for which meson is a good option).

[TingPing]

I realize its valuable but its not a realistic thing we can support. It is like wanting Meson to support building autotools projects. Cargo is its own independent and incompatible build system.

[zeenix]

@TingPing I would completely understand if you decide not to support this in meson but then either Rust support shouldn't be advertised at all or this limitation should be explicitly specified. As I mentioned, without this, Rust support is mostly useless.

[TingPing]

We have (basic) Rust support, we don't have Cargo support. It is sad that the Rust community only cares about Cargo and does not install shared libraries or have any tooling to collect information about installed libraries.

[tp-m]

While it is certainly "independent", It's not clear to me that it really is "incompatible". I see crates more like a special kind of subproject.

[zeenix]

@TingPing yeah, i know but fact remains that the support is useless for most uses so really nothing to advertise. I fully agree about Rust folks not doing things the standard way.

[TingPing]

I wasn't aware Meson was 'advertising' rust support other than it being documented. It is certainly basic.

[zeenix]

IMO basic isn't the correct term. Even the simplest of Rust projects will depend on external crates and meson can't handle those.

[tp-m]

I'd say let's focus on what a possible useful solution/implementation would require or could look like, and move the whole what should be advertised or not or whether something useful can be done without this feature or not thing elsewhere :)

[zeenix]

@tp-m Sure thing but it seemed like @TingPing is saying "we'll not be doing anything about this" in an earlier comment.

[nirbheek]

The main issue seems to be that cargo doesn't just download and manage dependencies, but also builds and links them? If there is no way around that, perhaps we can work with upstream to add a mode that only does the downloading bits.

[TingPing]

I would like Meson to be on par with Cargo in features. I don't know how doable it is to make Meson compatible with Cargo.

[TingPing]

@nirbheek Cargo is a full build system, it downloads dependences, builds them, links them, gets state from the environment, sets variables, conditional features, etc.

[mqudsi]

I think it's hard to appreciate the importance of cargo if you haven't developed in rust.

Here's the problem: unlike almost every other language, the rust "standard library" is purposely as small as possible and covers only what would be necessary to get a basic coding platform that works across multiple operating systems, sort of like a kernel abstraction layer. It doesn't provide much else. Early on in the rust project, code that code be stripped out of rust core and moved to an external package was removed from the standard library. In exchange, certain packages (available via cargo) are 'blessed', developed and maintained by the rust core developers, etc.

Cargo is more than a package manager, it's tightly built into the language itself. Unlike, say GCC or Clang++ with C/C++, rust has evolved with cargo from the start. You write test code inline with the rest of your code, knowing that cargo can pull it out and test it with cargo test. You use cargo to manage features in your code and in libraries you depend on (rough equivalent to make defines).

Unlike the situation with C and C++ and their various compilers, rust and cargo can't really be separated. Cargo provides dependency resolution, management, linking, and more, and like meson, has strongly structured contents (in Cargo.toml) that guide the build process, and differentiates between libraries and executables, etc.

Cargo is rich enough that it probably has more features than meson currently does. I don't think anyone would view it as a cop out at all if meson required cargo and used cargo instead of rustc for all rust interactions. I think it would be insane to try to reproduce the functionality of cargo in meson, and a recipe for disaster.

cargo already has project support, subproject support, source file discovery (provided via the language), and everything else meson would need to (selectively) build a project.

The problem is that simply shelling out to cargo means that dependencies are not defined in cargo.toml but rather in meson.build, but the solution to that is to have meson generate a valid cargo.toml file then run cargo. The structure of cargo.toml is both dumb enough and linear enough (with no dependency resolution there) that it would be a joke to write.

So, IMHO, the solution would be to have cargo carry out the work, but meson define the work that needs to be done. I would recommend considering cargo one compiler and rustc another - if people want a dependency-free, crate-free build they can use the rustc language/compiler and if they want the idiomatic rust experience, the cargo "language" and compiler. rustc would take arguments via the command line, while cargo would take arguments via a Cargo.toml file that can be generated at build time.

With regards to rebuilding, etc. I think relying on Cargo to do "the right thing" TM would not be unfair. Unlike C/C++, the artifacts of a cargo build are insane.

[TingPing]

Cargo is rich enough that it probably has more features than meson currently does.

I believe they expose roughly similar features and I think it is also possible to use only rustc while retaining those features. The --cfg flag seems to be our biggest missing feature currently.

the solution to that is to have meson generate a valid cargo.toml file then run cargo

Worth noting we would have to implement any support for toml that we need since we can't have dependencies.

I think there is no truely "good" solution as they made their ecosystem self-contained on purpose.

[mqudsi]

Thanks for the info. Toml files are just the modern day ini, there's nothing fancy enough to require a dependency (good on you for trying to keep those down to a minimum).

[TingPing]

toml seems to be ~1k LoC, maybe only a subset would be needed but annoying no matter.

[sdroege]

Unless you reimplement all of cargo or just call cargo, Rust support in meson is going to be useless. Everything is using lots of other crates and nobody is going to move all of them over to meson.

You can call rustc separately but that alone is not going to give you much. Consider cargo the "compiler" and not rustc ;)

You're going to have exactly the same problem with supporting any other language that has a tightly integrated build and dependency tool, and is not a scripting language. Haskell/cabal comes to mind here. Basically any language that did not want to go the C/C++ way of suffering wrt manual building in hundreds of different ways for dependencies.

If you want to become the universal build system, you have to integrate with the build tools of those languages in one way or another.

[sdroege]

toml seems to be ~1k LoC, maybe only a subset would be needed but annoying no matter.

Also toml is just the file format, a more well-defined INI style text format. You'll probably find Python libraries for that, but you'll need multiple times more code to do what cargo does. Also there's "cargo metadata" to parse the toml and do some more things already. But really, don't reimplement all that, it's a waste of time and you'll have to do the same for other languages and then always play catch up

[jpakkane]

Consider cargo the "compiler" and not rustc

But that is not going to work, because Cargo does its own dependency resolution and downloading. Suppose for example that we have a master project that uses a library called foo. Assume further that we want to use some Rust code that also uses foo via some sort of a Rust wrapper. The important thing to note here is that there must only be one version of foo and it must be the same one for both. If you just call into Cargo it will download and build its own crateified version of foo (at least this is what I have been told by Rust people that happens) but Meson does not know of this and thus downloads and builds its own version (which it must because things in the build dir are special).

Now we have two versions of foo scattered about. In the best possible case the binary is 2x bigger than it should be. In the worst case symbols from these get mixed up and that leads to debugging sprints from hell.

[sdroege]

The problem is that you don't even know if foo can be reused at all: it might have different, incompatible configurations for example. Cargo knows how to handle that though.

I would consider each Rust crate plus its dependencies one "unit" in meson, and let it output a cdylib (shared lib with only C API exported), staticlib (.a) or rlib (Rust shared lib). And that resulting artefact would then be used further in meson.

Cargo does not have to download things BTW, see "cargo vendor". But it does dependency resolution, build configuration (of dependencies and the top-level crate itself) and the actual build (which might involve not just rustc, but can be arbitrary commands via build.rs, including build of e.g. C code). And docs generation, unit testing, benchmarking, ...

[TingPing]

@sdroege Calling Meson's Rust support useless for not directly calling Cargo is like calling it's C support useless for not calling Autotools directly.

If a project uses a different build system it simply doesn't work as a subproject. The only part here that is frustrating is the Rust community doesn't install rlibs and doesn't have a system like pkg-config to sanely have external and versioned dependencies.

[mmstick]

I wanted to use Meson for handling non-Rust aspects of building and installing software, but without Cargo support, Meson is pretty much useless. My project is written 100% in Rust and Cargo works fine to produce a fully static binary with fully reproducible builds (thanks to the Cargo.lock file), but I need a means of checking if the dev box has mdbook installed, installing it if it isn't, generating the complete HTML documentation for the project, and offering a means of conveniently installing the project's documentation, building and installing plugins, and other miscellaneous files to the correct locations, and/or automating the process of packaging with certain package managers. Meson doesn't seem up to the task, so I guess I'll just use Ion to script this.

[sdroege]

@sdroege Calling Meson's Rust support useless for not directly calling Cargo is like calling it's C support useless for not calling Autotools directly.

Not really. Autotools is not really a standard for C and as you say instead things are installed as static or shared libraries in a defined way.

With Rust and lots of other languages (this is not really about Rust only!) the whole community decided on a tool for the job and everybody uses it. They're not going to switch to meson. So you would have things to be installed in a defined way, which in the case of Rust (and again, other languages) is also not the case and would have other problems. So as it is right now, the way how meson supports Rust is useless and there is a difference in philosophy that does not make both play well together unless one of them adapts. And again, this is not only Rust that works like that.

The general way of building things there is that you have a leaf-project and that is build as one unit with all its dependencies. For the dependencies the leaf-project could define specific configurations (which make the build results of those dependencies specific to the leaf-project and not reusable) and things like LTO can be performed in the end.

Having a defined way of installing build results would be nice to have, as would be shared libraries with a defined ABI (which would be less about code duplication than in C though, due to generics and monomorphisation, just like in C++). But that's not going to happen anytime soon in either of these languages as the immediate need is not there for anybody using them. Things work fine as-is.

[xclaesse]

And the command to compile, and recompile, an autotools project is also well known: 'make'. Let's add support for building autotools custom_target()/subproject()s next!

I totally agree we should make it for other build systems too, cargo is not special. I worked on that a couple years ago: #4321. In used a different idea, though.

Advantage: autotools projects, unlike cargo projects, actually produce .pc files today.

Yes, but sadly only a few generate -uninstalled.pc files. But I guess it's easier to patch them to generate the uninstalled pc file than to write a whole meson build definition.

(Problem: both cargo and make result in meson not being able to know when a project is up to date, so that gets proxied out to an external build system that gets run every time, not just when needed, and meson cannot tell the external build system when to force-remake something, nor have the external build system tell it when a meson artifact is out of date save by actually running said external build system and checking if the artifacts were updated.)

That's not really a problem. Meson just always invoke external build systems and rely on them to be no-op if nothing changed. In practice it's good enough.

[eli-schwartz]

Is it indeed good enough? A no-op build should complete quickly and preferably without verbose output, and that's certainly not the case for the average autotools project (on both counts).

Waiting on cargo to provide an introspection method like the one the current cmake module relies on, would ensure meson could integrate the necessary commands natively into build.ninja and achieve both those goals.

[xclaesse]

Is it indeed good enough? A no-op build should complete quickly and preferably without verbose output, and that's certainly not the case for the average autotools project (on both counts).

Well, if that's an issue then that project certainly should consider porting to meson/cmake. We can't do miracles.

Waiting on cargo to provide an introspection method like the one the current cmake module relies on, would ensure meson could integrate the necessary commands natively into build.ninja and achieve both those goals.

That probably means waiting a few decades, let's be honest, that just won't magically happen. And even that has serious issues, I'm not convinced at all the current cmake module way is better, totally fail at cross build, you still have the same issues to provide meson deps to cmake, etc.

[xclaesse]

TIL: Cargo actually generate depfiles and @nirbheek's meson module was already using them to avoid invoking cargo when not needed (with build_always_stale). So that's a problem already solved.

[amCap1712]

How can I use rust libraries with a project using meson build system and are there any examples to do so ? I am willing to use a script and a combination of both meson and cargo if needed.

[dreamer-coding]

I have found some documentation for crates.io web API. Hopefully this would help with adding support for Cargo crates.

Relevant document here.

[dcbaker]

I'm working on the 'reimplement cargo' route, like the way we handle cmake. It's coming up on my list of things to get back to.

[bogen85]

This issue has been open for over 5 years... I would like to see meson support external crates with Rust. On Fedora/RHEL/RHEL-derivatives there are many system packages for rust crates. I've found no clear way to even specify they be used in a meson project. Currently I'm still having to use Cargo instead of Meson, but I'd prefer meson as that is what I use for other languages.

Cargo is great, but Rust's over dependence on having to use it exclusively as the only build tool (??) makes Rust not feel as first class to me... I hoping that Meson can help Rust feel more first class as far as project building. True, thinks like AdaMake are baked into GNAT (Ada Tooling...) but at least AdaMake lets me use Ada libraries installed in system packages... I know, this is a discussion about Rust support in Meson... Which is difficult due to tunnel vision in Rust development caused by over dependence on Cargo exclusively for anything other than simple hello world program examples... (yes, I know one can specify simple crate paths directly, but that breaks down fast on more complex external crates)

[bogen85]

https://bugzilla.redhat.com/show_bug.cgi?id=1920959 that bug is closed due to Fedora 34 being end of life, but it highlights what I believe is at the core of the problem here... Rust's crate ecosystem does not fit well into established build systems (fedora system builds, meson, etc). And trying to make it fit is very painful. One "should" be able to install all the dependencies for a programming project just using system packages. (Yes, that might mean not being able to use some deps you would prefer to use, and having to cope with just being used what is available as an already installed packages). Rust crate ecosystem (which by default is tied to cargo) needs to updated/improved to "play well with others...." Because right now it does not... Hopefully that is addressed so that is not such a pain to get proper support into Meson. Cargo is overly dependent on an Internet connection... And the pulling in of unvetted dependencies... That in and of it self makes Rust painful to use for many software development projects. While problems make still trckle down, at least system packages have a lot more vetting applied to them.

Even when Meson gets proper external crate support, if it has to download deps off of github that is no good in my opinion. One needs to be able to use what they have already installed on their local system.

[eli-schwartz]

There are two factors here:

• integrating well with system library dependencies (C ABI, probably written in C)
• resolving rust rlib dependencies

Ideally both would work in packaged system dependencies form. In practice rust tooling (especially cargo) doesn't really "want" to do rlibs as anything other than built in vendored form. I don't know what the distribution system for that is supposed to look like, but perhaps we can invent one.

...

I do know that @dcbaker is working on this and is very motivated to do so as having a better crates story for Meson is a blocker for Mesa. The current mechanism being proposed will likely involve supporting only crates that don't use build.rs, which can be robustly translated into meson.build rules; I think the main blocker for this is waiting for python 3.11 and then figuring out the tomllib dependency for parsing Cargo.toml.

That would allow vendoring crates as checksummed subprojects (downloaded and distributed in dist tarballs, so no internet connection needed), building them via meson instead of cargo, and linking them to C libraries provided as subprojects which cargo can't know about, which is a definite and powerful improvement over the state of the art.

[dcbaker]

Yeah. I am still working toward being able to build rust crates in meson, including by vendoring the rust source in the meson produced dist tarball, as @eli-schwartz said. I've been working hard on paying down technical debt, as it's become a pain point in getting this working.

There are a bunch of issues with using rust libs as system dependencies:

1. there is no stable dynamic library ABI, which means that every change in the Rust compiler potentially requires rebuilding all of the rust packages on a system. Rust also heavily relies on templating, which makes dynamic libraries even more complicated, IIUC.
2. cargo doesn't support installing rlibs (which are static) system wide
3. pkg-config doesn't really support non-C-ABI dependencies @kaniini and I have talked about this a bit, and this is very much fixable

[sdroege]

1. there is no stable dynamic library ABI, which means that every change in the Rust compiler potentially requires rebuilding all of the rust packages on a system

(Also affecting 2.) Symbol names get a hash added to them that includes, among other things, the versions of all the crate's dependencies, the selected feature flags and IIRC also parts of the code related to the symbol. So changing any version of anything (including the compiler) or changing feature flags will require rebuilding everything on top of it.

You can set this hash yourself with some compiler flags IIRC but then you're really on your own with ensuring ABI compatibility.

If you want to do dynamic linking with Rust at this point you really need to build your whole set of libraries/binaries in one go.

Rust also heavily relies on templating, which makes dynamic libraries even more complicated, IIUC.

Not very different from C++ with header-only libraries though, however the generic code is part of the rlibs in non-source form and incompatible between compiler versions.

---

Linux distros (Fedora/Debian) are shipping Rust library crates in the -dev/-devel packages as source code for these reasons, and binaries statically link them in (those distro packages are basically used as a different way of vendoring / a separate cargo crates registry).

@dcbaker I assume you're aware of all that, just adding this extra information here because your comment makes the challenges sound simpler than they actually are in that regard :)

[LingMan]

@bogen85 Cargo doesn't need a network connection if you just tell it where to find the crates it needs locally. From the looks of it, Fedora rust packages build up a directory source at /usr/share/cargo/registry. From that link above: If you want to limit yourself to those, simply create a .cargo/config.toml with this content:

[source.my-local-fedora-source]
directory = "/usr/share/cargo/registry"

[source.crates-io]
replace-with = "my-local-fedora-source"

[net]
offline = true   

Untested and I don't have a Fedora system, so this might be a bit broken but should work in principle.
For a larger organization creating a vetted private crate repository would of course be a better option than a local dir.

[bogen85]

@LingMan

Untested and I don't have a Fedora system, so this might be a bit broken but should work in principle. For a larger organization creating a vetted private crate repository would of course be a better option than a local dir.

I tried this with a few crates in a simple project. It seems to work as intended. There is no reaching out to the Internet to do a build from scratch starting out with an empty ~/.cargo
Thanks!

Not every crate exists obviously but there are over 10K rust packages in Fedora 36...

Multiple variants exist for many crates:

rust-zmq+default-devel.noarch : High-level bindings to the zeromq library
rust-zmq+zmq_has-devel.noarch : High-level bindings to the zeromq library
rust-zmq-devel.noarch : High-level bindings to the zeromq library
rust-zmq-sys+default-devel.noarch : Low-level bindings to the zeromq library
rust-zmq-sys-devel.noarch : Low-level bindings to the zeromq library

In many cases, all the variants for a given crate can be installed at the same time, and I'm not completely sure what is up with that yet.

[carbotaniuman]

Is there any workaround/progress for this? I want to build some multi language code with this, but this is a major blocker.

[eli-schwartz]

@carbotaniuman Yes, the recent meson release can automatically convert a Cargo.toml into a meson subproject.

https://mesonbuild.com/Release-notes-for-1-3-0.html#automatic-fallback-to-cmake-and-cargo-subproject

This does not yet work for all cases, there are pending improvements to handle cargo features correctly as well as to automatically generate wraps by contacting https://crates.io when an existing subproject has a crate dependency which we don't have locally yet. Please follow #12363 for more details (and feel free to test out this experimental PR branch).

We are anxious to land it for meson 1.4.0 as there are several interested consumers that need it (and have been the driving force behind the recent focus on getting rust crates to work).