Rust backend

[rbuckland]

Preamble

This is a discussion thread to gather the requirements, ideas and wisdom for creating a development backend workflow for Rust.
There is some prior work already that we will continue on:

• Add Rust backend Issue 15775
• Initial PR by @tdyaas
• Design Doc by @tdyas

This is intended to be a bit of a brain dump and request for ideas/opinions/thoughts on creating a compilation and packaging workflow for the C and C++ backend(s). Currently only "firing" shell command is supported, no rust.

• ...src/python/pants/backend

This discussion thread is to flesh out the themes we would need, such that we can incrementally build out a Rust backend.
Base principle is to no reinvent any wheel, but let a rust project, live within in a wider polyglot monorepo (thing a Rust lib that generates Python, C and npm (webpack) libs)

Required Features

• Resource/Dependency Generation (bin or lib is output)
• Compilers/Toolchains
• Package Management
• Linting and Formatting
• Testing
• Compilation (inc multi-arch support)
• Caching
• Build Generators

Resource/Dependency Generation

• w.r.t. Dependency Inference:
  • Is reading the cargo.toml enough to determine what will be "spat out" - need to know the architectures being generated of course.

Compiler and Toolchain

The assumption would be that we can hook straight into cargo execution
Do we run rustc ?

Package Management

cargo.lock file exists, - need to look how it is handled With Python/pip,poetry and Java/maven today. This should be similar same, which has the best example ?

Linting and Formatting

run cargo fmt :-) ?

Testing

run cargo test :-) ?

Compilation

• same , run cargo.
• What about debug output formats etc
• multi architecture generation

Caching

need to know how the caching architecture works and how we seed it from a rust point of view.

Build Generators

• w.r.t. Dependency Inference:
• In theory this may just be, have extension points to that a call to "shell" can declare what will be output.
  An example (personally used) is auto generation of json-schema files, from the rust struct external data types.
  Then used in APIs.

Actions

• collate the above list into a clear set of requirements
• call out any dependencies or "solved by one action" between them
• identify any known gaps in Pants today that blocks us for beginning this work

[rbuckland]

from discussion on Slack - @benjyw mentioned "dependency inference"
so I may need to update the discussion list above (re-organise it) - dependency inference is a better term

e.g. :

• Dependency Inference
  • Compiled bin and lib files
  • Other Artifacts - webpacks, schema, protobuf files etc etc that Cargo may be in control of.

from: @benjyw - Oh, and the other missing bit is dep inference, unless that’s what you meant by “build generators”?

@rbuckland - kind of, not really
so dependency inference I can see as two areas.

• native language tool chain generation and dependency inference
  • it produces something - ramons-binary
  • we can determine the immediate and transitive dependencies that ramons-binary depends on
• "native tooling" is used to generate stuff, ramons-binary-api-dataclass-FooBar.schema.json

[tdyas]

Prior art:

• My initial attempt at a Rust backend: [WIP] [internal] introduce Rust backend with rustfmt support #15093
• Design document for what the Rust backend's target types should be.

[rbuckland]

awesome @tdyas thanks.

[tdyas]

Basically the work didn't make progress past the discussion on the design document around what the target types should be. The PR was able to invoke rustfmt. It relies on rustup to manage the Rust toolchain in use, which is an approach similar to how the Pants JVM backends rely on Coursier to manage JDK installations.

[benjyw]

Hi @rbuckland , wondering if you've had further thoughts on this?

[tgolsson]

Since it hasn't been mentioned there's an equivalent plugin for Bazel: https://github.com/google/cargo-raze. Might be interesting to look at that and see what parallels there are and what can be learned. From experience, one of the things I think raze does wrong is embedding one build-file per package into the repo. Huge source of noise. And it requires "non-native" tooling to work with by using cargo raze to generate Bazel build files.

[rbtcollins]

cargo-raze is deprecated; there's a newer thing built into rules_rust that is much much nicer. https://bazelbuild.github.io/rules_rust/crate_universe.html
Then you just do a bazel sync like so: https://bazelbuild.github.io/rules_rust/crate_universe.html#repinning--updating-dependencies

CARGO_BAZEL_REPIN=1 bazel sync --only=crate_index

This is not the nicest it could be - processing the Cargo workspace Cargo.lock directly would be superior I think.

[rbtcollins]

At this top of this discussion there is the concept of a rust thing in a monorepo, e.g. just building a lib.

We're a current user of Bazel, amongst other things, and for us a monorepo looks like a real polyglot situation: generally we build a docker image for a particular service, and we have dozens of services in a single repo; some are just Rust, some are just Go, some are just Python etc. A small number have both NPM and Python or some similar combination in one docker image; A small number have cross-language libraries being built.

I mention this just to say that the common case where a caching, hermetic, remote-build-capable build system would be useful to Rust users, is actually just building Rust docker images :). At least for us. Doing that fast, consistently, well, would allow making CI toolchains a heck of a lot nicer.

Bazel has rough edges particular related to C dependencies that Rust ends up bringing in. Like openssl_sys.

[huonw]

I noticed that cargo build currently supports a --build-plan argument, that outputs a JSON blob of the invocations cargo would be doing, including env vars etc.

For instance, running ./cargo +nightly build --build-plan -Z unstable-options in the pants repo starts like:

Details

{
  "invocations": [
    {
      "package_name": "peg-runtime",
      "package_version": "0.8.1",
      "target_kind": [
        "lib"
      ],
      "kind": null,
      "compile_mode": "build",
      "deps": [],
      "outputs": [
        "/Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/deps/libpeg_runtime-c5f23995afc28bca.rlib",
        "/Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/deps/libpeg_runtime-c5f23995afc28bca.rmeta"
      ],
      "links": {},
      "program": "rustc",
      "args": [
        "--crate-name",
        "peg_runtime",
        "--edition=2018",
        "/Users/huon/.cargo/registry/src/index.crates.io-6f17d22bba15001f/peg-runtime-0.8.1/lib.rs",
        "--error-format=json",
        "--json=diagnostic-rendered-ansi,artifacts,future-incompat",
        "--diagnostic-width=274",
        "--crate-type",
        "lib",
        "--emit=dep-info,metadata,link",
        "-C",
        "embed-bitcode=no",
        "-C",
        "split-debuginfo=unpacked",
        "-C",
        "debuginfo=2",
        "-C",
        "metadata=c5f23995afc28bca",
        "-C",
        "extra-filename=-c5f23995afc28bca",
        "--out-dir",
        "/Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/deps",
        "-L",
        "dependency=/Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/deps",
        "--cap-lints",
        "allow",
        "--cfg",
        "tokio_unstable"
      ],
      "env": {
        "CARGO": "/Users/huon/.rustup/toolchains/nightly-aarch64-apple-darwin/bin/cargo",
        "CARGO_CRATE_NAME": "peg_runtime",
        "CARGO_MANIFEST_DIR": "/Users/huon/.cargo/registry/src/index.crates.io-6f17d22bba15001f/peg-runtime-0.8.1",
        "CARGO_PKG_AUTHORS": "Kevin Mehall <[email protected]>",
        "CARGO_PKG_DESCRIPTION": "Runtime support for rust-peg grammars. To use rust-peg, see the `peg` crate.",
        "CARGO_PKG_HOMEPAGE": "",
        "CARGO_PKG_LICENSE": "MIT",
        "CARGO_PKG_LICENSE_FILE": "",
        "CARGO_PKG_NAME": "peg-runtime",
        "CARGO_PKG_REPOSITORY": "https://github.com/kevinmehall/rust-peg",
        "CARGO_PKG_RUST_VERSION": "",
        "CARGO_PKG_VERSION": "0.8.1",
        "CARGO_PKG_VERSION_MAJOR": "0",
        "CARGO_PKG_VERSION_MINOR": "8",
        "CARGO_PKG_VERSION_PATCH": "1",
        "CARGO_PKG_VERSION_PRE": "",
        "DYLD_FALLBACK_LIBRARY_PATH": "/Users/huon/projects/pantsbuild/pants/src/rust/engine/target/debug/deps:/Users/huon/.rustup/toolchains/nightly-aarch64-apple-darwin/lib:/Users/huon/.rustup/toolchains/nightly-aarch64-apple-darwin/lib:/Users/huon/lib:/usr/local/lib:/usr/lib"
      },
      "cwd": "/Users/huon/.cargo/registry/src/index.crates.io-6f17d22bba15001f/peg-runtime-0.8.1"
    },
...

I could imagine that this might be a path forward for integrating external dependencies via Cargo.tomls, although I imagine it's only one small building block (e.g. dep inference would still be required to determine which commands need to be rebuilt, and there's rather a lot of absolute paths in that output), and it's potentially not pants-y enough?

Relevant issues:

• Tracking issue for build plan generation (--build-plan) rust-lang/cargo#5579
• Should build-plans be deleted? rust-lang/cargo#7614 (!)