RFC: Deprecate the build= command

[tomaka]

The custom build command is a very wonky mechanism. It is not cross-platform, it assumes that the user has certain softwares installed on his system and available in his PATH (usually make/gcc, sometimes curl), it does not support cross-compilation, etc. Basically it is like having a makefile without a configure script.

This is especially problematic on Windows. Because of this command, a lot of Rust libraries depend on the user having MinGW installed (even though rustc no longer requires this), and some libraries simply can't be compiled on Windows, like glfw-rs. I have the sensation that all the efforts spent on improving support for Windows are worthless because of this.

I propose to deprecate the build command. At first, print a warning when compiling a crate that uses it, and in a second time remove support for it entirely.

Replacements

This command should be replaced by two complementary mechanisms.

Binary dependencies

Add a pre-build string entry in Cargo.toml. This string must in the format $package/$bin where $package is the name of a dependency in the dependencies array, and $bin is the name of a binary found in this dependency. The binary will be compiled and then run by Cargo similarly to the current build command.

The binary would have access to the environment variables that the build currently has access to, plus an additional one named COMMAND_CARGO_MANIFEST_DIR which contains the directory of the Cargo manifest of the dependency (instead of the manifest of the package where the command is being run).

Example:
/Cargo.toml:

[package]
name = "whatever"
authors = []
pre-build = "my-compiler/compile"

[dependencies.my-compiler]
path = "my-compiler"

/my-compiler/Cargo.toml:

[package]
name = "my-compiler"
authors = []

[[bin]]
name = "compile"

Why is this better than a regular build command?

• Instead of having a makefile run curl for example, you can instead write a binary that uses a Rust http library, which is more cross-platform because it does not require the user to have curl in his path.
• In situations where you would need a very complex pre-build script (for example invoking cmake (which requires determining the list of generators and invoking the right one)), instead of writing a difficult-to-maintain makefile that you copy-paste in multiple projects, you could simply use a Rust library.

Add native

Add a native array in Cargo.toml that specifies a list of non-rust libraries that must be present for this project to compile.

[[native]]
name = "GL"

[[native]]
name = "X11"

[[native]]
name = "Xxf86vm"

Cargo will process each entry in this order:

• Try to find a pre-compiled version of the library (if X is the library name, try to find libX.* and X.*) in $package_root/native/$target. If it finds a match, copy the file in $(OUT_DIR).
  For example if you are compiling for arm-linux-androideabi and require a library named hello, Cargo will look for native/arm-linux-androideabi/libhello.* and native/arm-linux-androideabi/hello.*.
• If we are not cross-compiling (ie. no --target option has been passed) and if we didn't find anything in the previous step, try to invoke pkg-config. If a result is found, pass the path of the library to rustc with the -L flag.
• If we didn't find anything in the previous steps, try to find a pre-compiled version of the library in $(HOME)/.rust. There is no additional step because rustc already looks into $(HOME)/.rust. Cargo must still check if the library is present.
• If nothing is found, print an error and stop. An additional flag --ignore-missing-native-libs could be added to Cargo in order to bypass this error and print a warning instead.

This system could be improved in the future by adding more entries to native, for example allowing the user to choose whether to link statically or dynamically.

[huonw]

Would an accurate summary of the first section essentially be: use Rust as a scripting language for Cargo?

(As a complete bystander this seems like a sensible approach, especially if we build up libraries that make the common tasks required there nice.)

[tomaka]

Would an accurate summary of the first section essentially be: use Rust as a scripting language for Cargo?

Yes :)

[mahkoh]

This sounds like a less powerful version of what I wanted for #427:

1. (Instead of/Additionally to) Cargo.toml, people can add a Configure.rs to their project root.
2. When you run cargo configure, and if Configure.rs has been modified since the last time, Configure.rs is compiled and run.
3. The binary can then generate a Cargo.toml file or drive the compilation itself, possibly depending on the flags passed to it.
4. This is made possible by automatically linking Configure.rs against a library provided by Cargo that allows you to do certain things with easy one-liners:
   • Download things
   • Call external programs
   • Detect native libraries, e.g., cargo::find("curl") will run pkg-config on Linux and do whatever you do on other systems on other systems
   • Use rustc

Add a native array in Cargo.toml that specifies a list of non-rust libraries that must be present for this project to compile.

Make them optional and you have #427.

[alexcrichton]

Thanks for taking the time to write this up, it looks quite nice!

Here's some thoughts I have on this:

• Only having one build command was never the long-term plan for cargo, this is just a stepping stone while the full feature set is implemented. Namely Add support for platform-specific build commands #552 is planned which will resolve some of the cross-platform issues you mentioned.
• It may also be worth mentioning some downsides to a "rust build system" (in addition to the upsides)
  • This does not yet exist, and it is likely very difficult to do well.
  • Writing a Makefile is quite familiar to anyone on unix, whereas this would be "one more build system"
  • Betting heavily on a nonexistent system can often be risky.
• Relying on common dependencies for build steps sounds amazing

The native section I think is much more nuanced and very tricky to get right. It's probably worth separating it out from the build command proposal. For example, it needs to address things like:

• Should the library be static or dynamic?
• What to do when pkg-config isn't available?
• What to do for different required libraries per-platform?

I believe @wycats and @carllerche have been working on a proposal for a modification to how cargo treats native libraries. I don't know the details of it yet but it is likely highly inter-related to this.

[tomaka]

The native section I think is much more nuanced and very tricky to get right.

I agree, but I really think that Cargo should have an explicit mechanism for this. Custom build commands should not be common, and shouldn't be used to build static libraries "just in case" the user doesn't have them.

I'd be happy to hear and work on another proposal for this.

Should the library be static or dynamic?

If the crate ships with a precompiled library, use it. The user chooses to put libraries in there, so he is not supposed to add multiple ones with the same name. An error should probably be printed if there are multiple matches.
The same applies for the $(HOME)/.rust step.

For pkg-config, since we are not cross-compiling, we can safely assume that the binary is going to be run on the same or a similar system. Linking dynamically shouldn't be a problem.

What to do when pkg-config isn't available?

Simply ignore that step.

We only have three situations:

• Your local unix system where pkg-config is available and you can use whatever library you have installed.
• Windows, where there is no mechanism to detect libraries and the common mechanism is to build everything, so we ignore that step.
• Cross-compiling, where any library available in the host system would probably be incompatible, so we ignore that step.

[vhbit]

@mahkoh

(Instead of/Additionally to) Cargo.toml, people can add a Configure.rs to their project root.
When you run cargo configure, and if Configure.rs has been modified since the last time, Configure.rs is compiled and run.

It actually doesn't work pretty well as:

1. It forces developer of a library to think about building and packaging on all possible platforms, which in most cases isn't his intention.
2. It forces user of a library to use what developer "provided" although it might be not what he needs, it might lag behind, it might a lot of other things.

Pre-built libraries definitely are much more flexible solution of the problem - when you need a lib on Linux most probably you will use {apt,yum,pacman,whatever} to install it rather than downloading and building it from sources.

[shadowmint]

This proposal would be a significant improvement on the adhoc solution for 'requires mingw' in #552

However, I would suggest slightly different semantics that make this slightly more consistent and plausible.

1. Make build rules simple and consistent through use of [[bin]]

To add a 'compile' step to a project, add a compiler binary to the project:

[[bin]]
name = "mylib-builder"
path = "build/bin.rs"

2. The 'build' target to a project may now optionally depend on a local [[bin]] target

Rather than removing 'build' (bad idea, breaks compatibility) we add the /target/ folder to the search PATH for invoking the build command.

If people want to keep using make to build their libraries, they can. It is bad to forbid people from doing this just because some people have strong opinions about platform compatibility (myself included. :P); the developer should always been in charge of make their own decisions.

3. Start building some strong third party cargo libraries to support invoking various build tools.

Having a .rs file that 'has to do everything' (ie. configure.rs) is probably a terrible idea.

However, I can easily imagine have a series of build libraries:

cargo-build-win32
cargo-build-cmake
cargo-build-autoconf
cargo-build-msbuild
etc.

As simple stand alone dependencies we can gain a rich build ecosystem that uses 3rd party tools correctly without having to fold all of that functionality into cargo itself, which will bloat cargo and never work properly.

We can now use per-platform configuration as normal in rust in the build binary without having to fold all of that stuff into cargo as well (which is why #552 is a bad idea).

By using external packages to manage the build wrappers (that essentially invoke cmake, msbuild, etc) the complexity that cargo has to look after is significantly reduced.

Resulting toml file

Might look something like:

[package]
name = "watch"
version = "0.1.1"
authors = [ "foo" ]
build = "watch-build"

[dependencies.glob]
git = "https://github.com/shadowmint/rust-glob"
tag = "0.1.6"

[dev-dependencies.cargo-cmake]
git = "https://github.com/rust-lang/cargo-cmake"
tag = "0.0.1"

[[bin]]
name = "watch-build"
path = "buiild/bin.rs"

[lib]
name = "watch"
path = "src/lib.rs"

With a build/bin.rs that does whatever it has to do, depending on the libraries:

#[cfg(win32)]
mod build {
  extern crate msbuild;
  use msbuild::builder;
  pub fn build() {
    var b = Builder::new();
    b.seekVisualStudio();
    b.setBuildRoot("...");
    b.build();
  }
}

#[cfg(unix)]
mod build {
  extern crate cmake;
  use cmake::CMake;
  pub fn build() {
    var b = CMake::new("root/");
    if b.configure() {
      b.build();
      #[cfg(test)]
      b.test();
      b.install();
    }
  }
}

fn main() {
  build::build();
}

There's so much richness you gain from this; small stand alone build tools can be maintained for build systems such as scons, cmake, etc. and the developer of a package can choose what to use. All of this stuff already exists.

It also allows for support of cfg options when invoking cargo.

Tangibly actionable items

It's also easy to implement:

1. Look in target/ for a binary first before calling the build command

2. Pass additional env variables to the build command

3. Make sure that certain 'bin' targets are compiled first (see below)

Unresolved

1. For this to work, (3) must be implemented somehow

So the build order is:

• build dependencies
• build 'build binaries'
• run build command
• build libraries
• build binaries

ie. Something is needed to mark certain [[bin]] targets as having build priority.

Perhaps [[dev-bin]] is appropriate for this? I'm not sure exactly the right thing for it.

Anyway, I'm 100% for this solution one way or another. The child-project that gets independently built has various other good properties too (does not clutter cargo.toml with build dependencies) but I prefer the above approach.

2. C dependency version conflicts / duplicate builds.

I've already encountered situations where rust libraries that build their own C dependencies results in multiple copies of a C dependency being built, causing issues.

One solution to this problem would be to definite a 'staging area' in target/ for dependent libraries to be put into, so that the build libraries could first look into that folder for a matching library before building their own copy.

This is a zero-cost effectively, as all it is is adding documentation and an extra env variable of some kind that the build libraries can use, but it is worth talking about.

(fwiw, the current system doesn't take this into account at all, and basically just assumes you already use the system library)

This would allow someone to write (for example) rust-libuv, which is a rust-free c wrapper package that provides a local cross platform libuv build without any 'specific' rust binding to it.

Other packages could then depend on this and implement their own custom or partial wrappers (eg. the file watch api) without requiring multiple builds of the same C library in different packages.

[shadowmint]

...and just to be clear, I'm not suggesting that cargo start supporting cargo-foo build libraries; absolutely not. This is the sort of thing that should fall to the 3rd party developer ecosystem as required.

I really think going down the path of forcing cargo to be responsible for searching for native binaries on every possible combination of system and library is a bad idea; it'd be massively complex to implement.

Rather, someone can write a rust-check-dyanmic-dep library that the build tool (if it assumes a dynamic library must be present) can use to query the OS and find a target for.

...or, you can rely on the native dependency resolution from the build tool if it supports it (eg. cmake)

[mahkoh]

@vhbit: I don't know where you get the idea it would build native libraries.

[vhbit]

@mahkoh from the second part of proposal (native section) + own cross-compiling experience. In most cases build is used just for preparing native (asm/C) libraries/wrappers before building Rust library which depends on them.

[shadowmint]

fwiw, prebuilt binaries are significantly more convenient in many cases, but also technically much more complex to manage correctly and securely (for example, do you invoke yum/apt to install a library? or download your own prebuilt copy from somewhere? does the prebuilt binary need to be signed? do you require root access to build a crate now?). Have a look at the complexity of wheel & pip.

I would strongly recommend that any plan for prebuilt binaries be deferred until after a solution for built sources is finalized.

I imagine the best solution would be for a package to search for a local copy, and then attempt to download a binary... and finally build its own if no local one is found; but I feel the complexity of that workflow is something that should split out of cargo into dependency libraries with their own custom binary repos and build tools as required.

If we want to standardize the resolution behavior and host binaries centrally at some point, thats something that can happen in a supported cargo-build-tools crate at a later date.

[vhbit]

I imagine the best solution would be for a package to search for a local copy

I'd say that the only thing I'd like to have is exactly this - flexibility in describing where to find native deps locally. Everything else (extra configure.rs or autodownloading and autobuilding) adds too much complexity. May be "later... someday..." :-)

[shadowmint]

@vhbit > I'd say that the only thing I'd like to have is exactly this ...

I know, but that's a naive approach to the problem.

On windows, it'll never be installed.

Additionally, on other relatively unix like system (osx say) without a package manager it will also never be installed. ...or it may be installed with different versions with different ABIs in multiple places (brew / port / fink / xcode)

If you're trying for a cross compile (say for android), it'll also never be installed.

Linking against system libraries only works on linux when you're building against your own local machine. It's a relatively useful solution for such a tiny subset of people I'm not actually sure it's worth explicitly supporting in cargo.

[vhbit]

It's a relatively useful solution for such a tiny subset of people I'm not actually sure it's worth explicitly supporting in cargo.

1. cargo is a Rust package manager/build tool. I'd be happy, but I can't see why it should solve problems of delivering native libraries (and/or resolving versioning issues). It's responsibility of platform tools and cargo should just build on top of that.
2. Cross platform native libraries are mostly required on terminating nodes (for example when you're terminating a tree of libraries into an application or a chain of commits into release). And those tasks are usually followed up by other platform specific tasks like packaging. Again this usually involves platform tools for packaging and so on.
3. When native libraries are required on internal nodes (for example a static library required during development process) - local one could be used no matter how delivered - built from source, used from package manager and so on.

I see cargo as a tool in a build chain which should just build all related to Rust. It should care only about this aspect of development process. Everything else should be done in other places, especially considering that except OS X/iOS[1] every other OS could be freely[2] and automatically[3] setup in a VM and the whole "terminating" process might be run there. Or might not. It depends on your build setup and allows flexibility by composing tools.

Delivering from one machine to all kind of platforms by just invoking cargo sounds extremely tempting and I hope someday it will be possible as number and maturity of available Rust libraries will increase (read after 1.0).

[1]: actually OS X works pretty fine in VM too, but legally you can run it only on Apple hardware.
[2]: there is a free Windows VM package for web developers. It is active 30 days, but that should be enough for building and packaging any kind of project.
[3]: at least VirtualBox provides pretty good scripting for creating and controlling VM, so just get a script for installing all prerequisites, starting build process and have fun.