Explore sandboxed build scripts

[nikomatsakis]

Metadata
Owner(s)	@weihanglo
Team(s)	cargo, compiler
Goal document	2024h2/sandboxed-build-script

Summary

Explore different strategies for sandboxing build script executions in Cargo.

Tasks and status

• Design (@weihanglo)
• Discussion and moral support (cargo )
• Security reviews ()
• Standard reviews (cargo )
• Collaboration with GSoC proc-macro project (compiler )
• Summary of experiments or RFC (@weihanglo)

[nikomatsakis]

This issue is intended for status updates only.

For general questions or comments, please contact the owner(s) directly.

[weihanglo]

Key developments:

Have been looking into different sandbox runtime choices. Here is a simple version of the comparison of three potential choices:

• eBPF
  • Required privilege
    • When installing: need other capability for like network monitoring CAP_PERFMON CAP_NET_ADMIN
    • When running: CAP_SYS_ADMIN or CAP_BPF for loading eBPF programs, unless unprivileged eBPF is enabled
    • Most Linux distributions don't allow unprivileged users to run eBPF programs.
  • Cross-platform
    • Linux: ✅
    • Windows: 🚧 https://github.com/microsoft/ebpf-for-windows
    • Darwin: ❓ https://www.reddit.com/r/eBPF/comments/y1whyh/plans_for_ebpf_on_macos/
  • Abilities
    • Process spawn: ✅
    • Network access: ✅
    • Filesystem access: ✅
  • How to use: leverage LLVM or any compiler to build eBPF bytecode
  • Security
    • Check capabilities when loading a eBPF program
• OCI spec compliant runtime
  • Required privilege
  • Cross-platform: ✅
  • Abilities
    • Process spawn: ✅ varies on different platforms
    • Network access: ✅ varies on different platforms
    • Filesystem access: ✅ via mounts in config.json
  • How to use: usually invoke a standalone binary (youki), but some are embeddable (northstar).
  • Security
  • Container technologies are based on system virtualization and isolation mechanisms, suchas cgropu or Hyper-V. These may bring large footprints (Can you imagine cargo run ubuntu:latest to run docker image!?), and is not really portable.
  • Well-defined (it has a specification!)
• WASI
  • Required privilege
    • Usually not required.
  • Cross-platform: ✅
  • Abilities
    • Process spawn: ❓ Unknown posix_spawn/CreateProcess API WebAssembly/WASI#414.
    • Network access: 🚧 Phase 3 (implementation) https://github.com/WebAssembly/wasi-sockets
    • Filesystem access: 🚧 Phase 3 (implementation) https://github.com/WebAssembly/wasi-filesystem.
  • WASI is designed exactly for sandbox use cases.
  • Might not be configurable enough; it depends on runtime implementations and/or Wasi-libc. Usually permissions are granted via CLI options, for example wasmtime --dir.

There are prior research on cross-over between each of these options. I've been busy these two weeks. Will update a more detail post for prior arts afterward.

The biggest challenge I am seeing now is spawning external processes. Most build script usages invoke some external binaries, like pkg-config for building *-sys crates, or protoc for generating protobuf bindings. If process spawning is that common, we need to find a way to provide a fine-grained permission granting scheme. I don't want it to see an “all-or-nothing” mechanism when process spawning is needed.

The other huge headache is setting library search paths. We cannot know every possible path of system libraries ahead of time, but we need to grant access to the runtime.

Blockers:

None.

Help wanted:

None.

[weihanglo]

Having a family urgency. I will be back after RustConf.

[weihanglo]

Key developments:

Building a workable version of wasm-based build script (not yet done). There are some technical difficulties. Not blocking but need to be addressed.

• Build script is assumed to build for host platform everywhere in Cargo's codebase, which is hard to abstract away or hack around.
• wasm32-wasip1 is not shipped by default; we need to rustup target add wasm32-wasip1 to build wasm.
• Also, Cargo learns info of target platforms on-demand if --target flag is provided. Not providing --target wasm32-wasip1 means no target information available. AFAIK, this is already an issue of artifact-dependencies, e.g., Panic when combining -Zbuild-std with artifact dependencies cargo#10444 and Panic when depending on a package with artifact dependencies cargo#11260.

In order to make the development independent of Cargo, RUSTC_WRAPPER was the first approach I tried, though it still lacks some extension point in Cargo:

• There is no way to intercept build script execution, so we need to hack into Cargo itself. RUSTC_WRAPPER can only intercept rustc invocations.
• target.runner is available but for cargo test and cargo run only.

Some compatibility issues came up when integrating with Cargo:

• How to handle build script override.
• artifact-dependencies can compile build script to any other platform, making it challenging to handle.

Blockers:

None.

Help wanted:

None.