Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cargo extension for compiling Rust to MASM #61

Merged
merged 22 commits into from
Dec 12, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
367bb3c
chore: draft `compile` command options using clap
greenhat Nov 20, 2023
f09a893
test: draft `compile` command using the `midenc_driver`, add test
greenhat Nov 20, 2023
e706024
test: get target dir from CARGO_TARGET_DIR env var when locating Wasm…
greenhat Nov 10, 2023
66a3d32
fix: emit Session artifact in CodegenStage, don't require Session.mat…
greenhat Nov 21, 2023
134fce3
test: check target folder exists after build, not before
greenhat Nov 10, 2023
2db01ff
chore: print the progress of the compilation
greenhat Nov 13, 2023
555051b
refactor: make `cargo_ext:compile` return `anyhow::Result`
greenhat Nov 13, 2023
5ad3645
refactor: use cargo-metadata to get Wasm artifacts
greenhat Nov 20, 2023
e108242
chore: add README.md to cargo-ext
greenhat Nov 14, 2023
58d3309
feat: add `new` command that generates a new Miden project from a tem…
greenhat Nov 20, 2023
824a8eb
fix: fix build after the rebase
greenhat Nov 20, 2023
62adf34
fix: switch to https://github.com/0xPolygonMiden/rust-templates
greenhat Nov 21, 2023
af32c71
fix build after rebase
greenhat Nov 21, 2023
5a8a6ed
refactor: switch from emiting MASM in CodegenStage, and switch to out…
greenhat Nov 21, 2023
91a8b3c
test: in cargo extension test create output folder if not exist
greenhat Nov 21, 2023
29ffb83
refactor: move cargo-ext to tools/cargo-miden
greenhat Dec 5, 2023
426096c
chore: remove miden-cargo-extension from workspace deps
greenhat Dec 5, 2023
5f0892f
remove `next_display_order` option in `Command`
greenhat Dec 5, 2023
a69711b
fix: rename `compile` cargo extension command to `build`; imports cle…
greenhat Dec 5, 2023
253d491
refactor: remove `path-absolutize` dependency
greenhat Dec 5, 2023
6e563ef
feat: revamp cargo-miden to pass the unrecognized options to cargo,
greenhat Dec 7, 2023
769d983
refactor: make `WasmTranslationConfig::module_name_fallback` non-opti…
greenhat Dec 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,254 changes: 1,220 additions & 34 deletions Cargo.lock

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@ members = [
"frontend-wasm",
"tests/rust-apps/*",
"tests/integration",
"cargo-ext",
bitwalker marked this conversation as resolved.
Show resolved Hide resolved
]
exclude = ["tests/rust-apps-wasm"]
exclude = ["tests/rust-apps-wasm", "cargo-ext/tests/data"]

[workspace.package]
version = "0.1.0"
Expand Down Expand Up @@ -77,6 +78,7 @@ midenc-compile = { path = "midenc-compile" }
midenc-driver = { path = "midenc-driver" }
midenc-session = { path = "midenc-session" }
miden-integration-tests = { path = "tests/integration" }
miden-cargo-extension = { path = "cargo-ext" }
bitwalker marked this conversation as resolved.
Show resolved Hide resolved


[profile.dev]
Expand Down
35 changes: 35 additions & 0 deletions cargo-ext/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
[package]
name = "cargo-miden"
version.workspace = true
rust-version.workspace = true
authors.workspace = true
description.workspace = true
repository.workspace = true
homepage.workspace = true
documentation.workspace = true
categories.workspace = true
keywords.workspace = true
license.workspace = true
readme.workspace = true
edition.workspace = true
publish.workspace = true
autotests = false # disable autodiscovery of tests

[[bin]]
name = "cargo-miden"

[[test]]
name = "integration"
path = "tests/mod.rs"

[dependencies]
midenc-compile.workspace = true
midenc-session.workspace = true
miden-diagnostics.workspace = true
clap.workspace = true
anyhow.workspace = true
cargo_metadata = "0.18"
cargo-generate = "0.18"
path-absolutize = "3.1.1"

[dev-dependencies]
50 changes: 50 additions & 0 deletions cargo-ext/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Miden Cargo Extension

This crate provides a cargo extension that allows you to compile Rust code to Miden VM MASM.

## Installation

To install the extension, run:

```bash
cargo install cargo-miden
```

## Requirements

Since Rust is first compiled to Wasm, you'll need to have the `wasm32-unknown-unknown` target installed:

```bash
rustup target add wasm32-unknown-unknown
```

## Usage

### Getting help
To get help with the extension, run:

```bash
cargo miden
```

Or for help with a specific command:

```bash
cargo miden <command> --help
```

### Creating a new project
To create a new Miden VM project, run:

```bash
cargo miden new <project-name>
```

### Compiling a project
To compile a Rust crate to Miden VM MASM, run:

```bash
cargo miden compile -o <output-file>
```

Without any additional arguments, this will compile the library target of the crate in the current directory.
48 changes: 48 additions & 0 deletions cargo-ext/src/cli_commands.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
use std::path::PathBuf;

use clap::Parser;
use clap::Subcommand;
use midenc_session::TargetEnv;

#[derive(Parser, Debug)]
#[command(name = "cargo")]
#[command(bin_name = "cargo")]
pub enum CargoCli {
Miden(MidenArgs),
}

#[derive(Parser, Debug)]
#[command(name = "miden")]
#[command(bin_name = "cargo miden")]
#[command(about = "Cargo command for developing Miden projects", long_about = None)]
pub struct MidenArgs {
#[command(subcommand)]
pub command: Commands,
}

#[derive(Debug, Subcommand)]
pub enum Commands {
/// Compile the current project to MASM
#[command(next_display_order(10), name = "compile")]
bitwalker marked this conversation as resolved.
Show resolved Hide resolved
Compile {
bitwalker marked this conversation as resolved.
Show resolved Hide resolved
/// The target environment to compile for
#[arg(long = "target", value_name = "TARGET", default_value_t = TargetEnv::Base, display_order(2))]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can change long = "target" to just target, and remove the display_order flag (unless there is a specific need for it that isn't apparent from the context).

target: TargetEnv,

/// Tells the compiler to produce an executable Miden program from the binary target or a library from the lib target if not specified
bitwalker marked this conversation as resolved.
Show resolved Hide resolved
#[arg(long = "bin-name", display_order(3))]
bin_name: Option<String>,

/// Output directory for the compiled MASM file(s)
bitwalker marked this conversation as resolved.
Show resolved Hide resolved
#[arg(
long = "out-dir",
value_name = "FOLDER",
id = "output-folder",
display_order(6)
)]
output_folder: PathBuf,
},
/// Scaffold a new Miden project at the given path
#[command(next_display_order(10), name = "new")]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can just be #[command] the rest is implied by default (and I don't think next_display_order is needed here).

New { path: PathBuf },
}
135 changes: 135 additions & 0 deletions cargo-ext/src/compile.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
use anyhow::bail;
use cargo_metadata::Message;
use std::path::PathBuf;
use std::process::Command;
use std::process::Stdio;
use std::sync::Arc;

use anyhow::Context;
use miden_diagnostics::Verbosity;
use midenc_session::InputFile;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should combine all these midenc_session imports into a single use since they are all from the same namespace.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm so used to "rust-analyzer.imports.granularity.group": "item" to avoid the merge conflicts, so I forgot to turn it off, given that our codebase employs a different import style. Fixed.
Although I'd strongly consider switching to item since it's more rebase/merge friendly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I generally agree with you, and I'm not really a fan of the approach used by the other Miden projects, so I've compromised by using something that is more of a blend: if it's in the same namespace, combine it, e.g. use a::b::Foo; and use a::b::Bar would get combined, but use a::b::Foo and use a::c::Baz would be separated. That generally avoids most conflicts due to changes in imports, while collapsing them considerably.

If this was my own project, I'd probably switch to grouping by item, but I wanted to at least make the attempt to follow the Miden guidelines to some degree here.

If there's a config file we can add to the project root that sets that rust-analyzer config option locally for the project, feel free to add it. Definitely annoying to have to change it globally just for the Miden projects, so if there's any way to avoid that, I'm open to it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. It seems this import granularity is Module. I made an issue - #78

use midenc_session::OutputFile;
use midenc_session::OutputType;
use midenc_session::OutputTypeSpec;
use midenc_session::OutputTypes;
use midenc_session::ProjectType;
use midenc_session::Session;
use midenc_session::TargetEnv;

pub fn compile(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use cargo_metadata::MetadataCommand to get metadata about the current workspace, and use that metadata to drive the build. In particular, we can use package metadata section, to configure defaults for the project. Most importantly, we'll need to do that anyway in order to get configuration for cargo miden (and we may also want to access cargo component configuration) from Cargo.toml.

For example, a single crate project we can use cargo_metadata::MetadataCommand to get the crate metadata - project name, targets, etc.; and use that to determine the default project type being built, the output file name we expect, and so on. We can extend this, or override defaults using package metadata, e.g. [package.metadata.miden].

On the other hand, for a workspace consisting of multiple crates, we can use the workspace metadata section ( e.g. [workspace.metadata.miden]) to specify which projects in the workspace are Miden projects, and use cargo_metadata::MetadataCommand to obtain metadata for all crates in the workspace, and then get the cargo_metadata::Package information about the crates specified in our package metadata section. By default, cargo miden build in this situation would build all Miden projects using their default configuration (using separate invocations of compile internally).

To illustrate what I mean:

# Cargo.toml
[workspace]
members = ["crates/*"]

[workspace.metadata.miden]
components = ["crates/foo", "crates/bar"]

# crates/foo/Cargo.toml
[package]
name = "foo"

[[bin]]
name = "foo-a"

[[bin]]
name = "foo-b"

[lib]
crate-type = ["cdylib", "rlib"]

[package.metadata.miden]
# if unspecified, the lib target is the default
default-target = "foo-a"

# crates/bar/Cargo.toml
[package]
name = "bar"

# This crate has no Miden metadata, because the default target is implied, and no other configuration is needed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! We'll need options to specify dependencies, etc.. Still, I think components and default-target options will add extra complexity for the newcomers and might do more harm than good. Given that they are not strictly necessary for the build process, we should get back to them later when we have a clear picture of the smart contract development workflow (crate structure, testing, etc.).
To drive the build process, I'd instead take the cargo-component approach. See my comment - #61 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still, I think components and default-target options will add extra complexity for the newcomers and might do more harm than good

To be clear, those are not needed with a default project setup, only for more complex workspaces which contain a blend of Miden and non-Miden crates. For simple cases, we can just assume all crates are Miden crates, and use the default targets implied by the Cargo configuration. It is very unlikely that most crates will have anything other than a single lib or bin target anyway.

target: TargetEnv,
bin_name: Option<String>,
output_folder: &PathBuf,
) -> anyhow::Result<()> {
// for cargo env var see https://doc.rust-lang.org/cargo/reference/environment-variables.html
let mut cargo_build_cmd = Command::new("cargo");
cargo_build_cmd
.arg("build")
.arg("--release")
.arg("--target=wasm32-unknown-unknown");
let project_type = if let Some(ref bin_name) = bin_name {
cargo_build_cmd.arg("--bin").arg(bin_name.clone());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So both --lib and --bin are implied by default with cargo build, depending on whether the crate has a default target (--bin is implied if there is a src/main.rs, and --lib if src/lib.rs). So we only need to obtain the default target from the Cargo metadata of the crate being compiled to detect the appropriate ProjectType.

If we want to allow specifying a target, we should follow the conventions of cargo build I think, to avoid having different defaults than expected.

Copy link
Contributor

@bitwalker bitwalker Dec 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: See #61 (comment), but my point about following the conventions of cargo build still holds in terms of choosing defaults IMO.

Copy link
Contributor Author

@greenhat greenhat Dec 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to allow specifying a target, we should follow the conventions of cargo build I think, to avoid having different defaults than expected.

I like the idea of mirroring cargo build options and semantics! It got me thinking about taking the approach of cargo component for building the Wasm component - just pass all the options to cargo and pick up Wasm files afterward. See https://github.com/bytecodealliance/cargo-component/blob/87949555d067e92a1872c03daa272794a6c0f6a5/src/lib.rs#L132-L180
I think we can take the same approach for our build(compile) command. This means for our build command, we'll have only --miden-target option for VM vs. rollup (I'm not even sure we need that), and the rest will be passed to cargo build (or rather cargo component build) as is adding --target wasm32-wasi if it's not present. I believe the Wasm component binary alone would suffice to figure out if Miden program or library is being built.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm fine with following cargo component here, though I think it remains to be seen whether our more complex build process forces us to be more restrictive than they are. That said, it feels like a better starting point for sure, and will probably get us quite far before we need to do anything special.

ProjectType::Program
} else {
ProjectType::Library
};
println!("Compiling Wasm with cargo build ...");
let mut child = cargo_build_cmd
.arg("--message-format=json-render-diagnostics")
.stdout(Stdio::piped())
.spawn()
.with_context(|| {
format!(
"Failed to execute cargo build {}.",
cargo_build_cmd
.get_args()
.map(|arg| format!("'{}'", arg.to_str().unwrap()))
.collect::<Vec<_>>()
.join(" ")
)
})?;
let reader = std::io::BufReader::new(child.stdout.take().unwrap());
let mut wasm_artifacts = Vec::new();
for message in cargo_metadata::Message::parse_stream(reader) {
match message.context("Failed to parse cargo metadata")? {
Message::CompilerArtifact(artifact) => {
// find the Wasm artifact in artifact.filenames
for filename in artifact.filenames {
if filename.as_str().ends_with(".wasm") {
wasm_artifacts.push(filename.into_std_path_buf());
}
}
}
_ => (),
}
}
let output = child.wait().expect("Couldn't get cargo's exit status");
if !output.success() {
bail!("Rust to Wasm compilation failed!");
}

if wasm_artifacts.is_empty() {
match project_type {
ProjectType::Library => bail!("Cargo build failed, no Wasm artifact found. Check if crate-type = [\"cdylib\"] is set in Cargo.toml"),
ProjectType::Program => bail!("Cargo build failed, no Wasm artifact found."),
}
}
if wasm_artifacts.len() > 1 {
bail!(
"Cargo build failed, multiple Wasm artifacts found: {:?}. Only one Wasm artifact is expected.",
wasm_artifacts
);
}
let wasm_file_path = wasm_artifacts[0].clone();
match project_type {
ProjectType::Program => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will have enough project metadata with the other changes I've mentioned that we can just filter the CompilerArtifact messages. We either have a matching artifact or we don't have any once the build finishes.

Copy link
Contributor Author

@greenhat greenhat Dec 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we'll go the route I outlined in #61 (comment), I think we might get away without specifying ProjectType to midenc or add an AutoDetect variant. If not, we can peek inside the Wasm component binary to determine what we're building. We may need this in other parts of our ecosystem anyway.

let bin_name = bin_name.unwrap();
if !wasm_file_path.ends_with(format!("{}.wasm", bin_name)) {
bail!(
"Cargo build failed, Wasm artifact name {} does not match the expected name '{}'.",
wasm_file_path.to_str().unwrap(),
bin_name
);
}
}
ProjectType::Library => (),
}

if !output_folder.exists() {
bail!(
"Output folder '{}' does not exist.",
output_folder.to_str().unwrap()
);
}

println!(
"Compiling '{}' Wasm to '{}' directory with midenc ...",
wasm_file_path.to_str().unwrap(),
&output_folder.as_path().to_str().unwrap()
);
let input = InputFile::from_path(wasm_file_path).context("Invalid input file")?;
let output_file_folder = OutputFile::Real(output_folder.clone());
let output_types = OutputTypes::new(vec![OutputTypeSpec {
output_type: OutputType::Masm,
path: Some(output_file_folder.clone()),
}]);
let cwd = std::env::current_dir().context("Failed to get current working directory")?;
let options = midenc_session::Options::new(cwd)
// .with_color(color)
.with_verbosity(Verbosity::Debug)
// .with_warnings(self.warn)
.with_output_types(output_types);
let session = Arc::new(
Session::new(
target,
input,
Some(output_folder.clone()),
None,
None,
options,
None,
)
.with_project_type(project_type),
);
midenc_compile::compile(session.clone()).context("Wasm to MASM compilation failed!")
}
5 changes: 5 additions & 0 deletions cargo-ext/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
mod compile;
mod new_project;

pub use compile::compile;
pub use new_project::new_project;
24 changes: 24 additions & 0 deletions cargo-ext/src/main.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
use anyhow::Context;
use cargo_miden::compile;
use cargo_miden::new_project;
use clap::Parser;
use cli_commands::CargoCli;
use cli_commands::Commands;

mod cli_commands;

fn main() -> anyhow::Result<()> {
let args = match CargoCli::parse() {
CargoCli::Miden(args) => args,
};

match args.command {
Commands::Compile {
target,
bin_name,
output_folder,
} => compile(target, bin_name, &output_folder)
.context(format!("Failed to compile {}", target)),
Commands::New { path } => new_project(path).context("Failed to scaffold a new project"),
}
}
41 changes: 41 additions & 0 deletions cargo-ext/src/new_project.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
use std::path::PathBuf;

use anyhow::Context;
use cargo_generate::GenerateArgs;
use cargo_generate::TemplatePath;

pub fn new_project(path: PathBuf) -> anyhow::Result<()> {
let name = path
.file_name()
.ok_or_else(|| {
anyhow::anyhow!("Failed to get the last segment of the provided path for the project name")
})?
.to_str()
.ok_or_else(|| {
anyhow::anyhow!("The last segment of the provided path must be valid UTF8 to generate a valid project name")
})?
.to_string();

let generate_args = GenerateArgs {
template_path: TemplatePath {
git: Some("https://github.com/0xPolygonMiden/rust-templates".into()),
auto_path: Some("library".into()),
..Default::default()
},
destination: path
.parent()
.map(|p| {
use path_absolutize::Absolutize;
bitwalker marked this conversation as resolved.
Show resolved Hide resolved
p.absolutize().map(|p| p.to_path_buf())
})
.transpose()
.context("Failed to convert destination path to an absolute path")?,
name: Some(name),
force_git_init: true,
verbose: true,
..Default::default()
};
cargo_generate::generate(generate_args)
.context("Failed to scaffold new Miden project from the template")?;
return Ok(());
}
24 changes: 24 additions & 0 deletions cargo-ext/tests/compile.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
use crate::utils::get_test_path;
use cargo_miden::compile;
use midenc_session::TargetEnv;
use std::env;
use std::fs;

#[test]
fn compile_template() {
let restore_dir = env::current_dir().unwrap();
let test_dir = get_test_path("template");
// dbg!(&test_dir);
env::set_current_dir(&test_dir).unwrap();
let masm_path_rel = "target";
let output_folder = test_dir.join(masm_path_rel);
if !output_folder.exists() {
fs::create_dir_all(&output_folder).unwrap();
}
compile(TargetEnv::Base, None, &output_folder).expect("Failed to compile");
env::set_current_dir(restore_dir).unwrap();
let expected_masm_path = output_folder.join("miden_wallet_lib.masm");
assert!(expected_masm_path.exists());
assert!(expected_masm_path.metadata().unwrap().len() > 0);
fs::remove_file(expected_masm_path).unwrap();
}
7 changes: 7 additions & 0 deletions cargo-ext/tests/data/template/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading