Cargo extension for compiling Rust to MASM #61

greenhat · 2023-11-08T15:13:19Z

Close #60

This PR introduces a cargo extension in the cargo-miden crate, which supports the following commands:

miden build - compiles the selected target to MASM
miden new - creates a new project from a template. The template is downloaded from a git repository.

Keep in mind that this cargo extension is planned to be the cargo-component
replacement when it comes to building/publishing/etc the Wasm/Miden components.
This should explain some implementation details. The overarching idea is
to introduce specific to Miden subcommands (e.g., new) and pass the
rest of the subcommands to cargo (e.g., build) and process the produced artifacts.

If the wasm32-was target is not present, it will be installed.

midenc-driver/src/commands/compile/stages/rewrite.rs

cargo-ext/src/compile.rs

greenhat · 2023-11-15T12:30:08Z

@bobbinth Could you please create a new repo miden-project-template (or another name), for the template of the Miden project to be used by the cargo extension when creating a new project with cargo miden new command? I temporarily made https://github.com/greenhat/miden-project-template, but I cannot transfer it because github, for some reason, requires that I have a right to create a repo in the target organization. I'll copy the repo content ASAP. Thank you!

bobbinth · 2023-11-16T23:55:54Z

@bobbinth Could you please create a new repo miden-project-template (or another name), for the template of the Miden project to be used by the cargo extension when creating a new project with cargo miden new command?

What kind of projects would this be for? For example, if this is to create smart contracts (i.e., account code), it should probably be called miden-account-template, and then in the future we'd have also miden-note-template (or something like this).

One other question: do we need to create a separate repo per command? Or could we create one repo (e.g., miden-rust-templates) and then put all templates into that repo?

bitwalker · 2023-11-17T02:00:14Z

What kind of projects would this be for? For example, if this is to create smart contracts (i.e., account code), it should probably be called miden-account-template, and then in the future we'd have also miden-note-template (or something like this).

I think this is probably just for a simple Rust project template initially.

One other question: do we need to create a separate repo per command? Or could we create one repo (e.g., miden-rust-templates) and then put all templates into that repo?

We need a repo per template I believe, so if we have three project templates, that's three separate repos. I'm not sure if we can put all templates in a single repo or not, @greenhat would be able to say for sure.

greenhat · 2023-11-17T12:47:33Z

What kind of projects would this be for? For example, if this is to create smart contracts (i.e., account code), it should probably be called miden-account-template, and then in the future we'd have also miden-note-template (or something like this).

I think this is probably just for a simple Rust project template initially.

Yes, so far, it's just compiling a lib target to the MASM module. No smart contract stuff yet. I assume we want to jump on account code right away and get back to dealing with generic Miden VM programs/libs later. So, I either evolve this template or create a new one for the account code.

One other question: do we need to create a separate repo per command? Or could we create one repo (e.g., miden-rust-templates) and then put all templates into that repo?

We need a repo per template I believe, so if we have three project templates, that's three separate repos. I'm not sure if we can put all templates in a single repo or not, @greenhat would be able to say for sure.

Good question! cargo-generate supports multiple templates in a single repo. And although initially, I wanted to have a single repo per template, I cannot find a good reason for it. It's not public facing, meaning no one will clone it, and the code cannot be built by cargo anyway. I think it makes sense to have a single repo for all templates. miden-rust-templates sounds good to me.

bobbinth · 2023-11-17T23:48:03Z

@greenhat - I've created rust-templates and you should have write access to it. Let me know if anything else is needed.

compiling a template project

… artifact

…ches in ApplyRewritesStage Otherwise, ApplyRewritesStage crashes when arg matches are not provided in Session.

…plate

…put folder in cargo extension Add WasmTranslationConfig::module_name_fallback to use if Wasm module doesn't specify a name in the custom section. In `Session::emit` if the path is a directory, emit to a file in that directory named after a compiled item(module) name.

greenhat · 2023-11-21T15:17:29Z

@bitwalker ready for review

bitwalker

Nice work!

There's a few things I'd like to change about the way the extension works, and a few cosmetic things, but overall this looks good to me. I think we can probably have this merged tomorrow, but let me know if some of the changes I've suggested will complicate that and we can discuss - but I can either pick up where you leave off tomorrow, or we can just wait an extra day or whatever.

Cargo.toml

cargo-ext/src/cli_commands.rs

bitwalker · 2023-12-04T18:57:24Z

cargo-ext/src/cli_commands.rs

+    #[command(next_display_order(10), name = "compile")]
+    Compile {
+        /// The target environment to compile for
+        #[arg(long = "target", value_name = "TARGET", default_value_t = TargetEnv::Base, display_order(2))]


You can change long = "target" to just target, and remove the display_order flag (unless there is a specific need for it that isn't apparent from the context).

cargo-ext/src/cli_commands.rs

bitwalker · 2023-12-04T19:54:18Z

cargo-ext/src/compile.rs

+    }
+    let wasm_file_path = wasm_artifacts[0].clone();
+    match project_type {
+        ProjectType::Program => {


I think we will have enough project metadata with the other changes I've mentioned that we can just filter the CompilerArtifact messages. We either have a matching artifact or we don't have any once the build finishes.

If we'll go the route I outlined in #61 (comment), I think we might get away without specifying ProjectType to midenc or add an AutoDetect variant. If not, we can peek inside the Wasm component binary to determine what we're building. We may need this in other parts of our ecosystem anyway.

cargo-ext/src/cli_commands.rs

cargo-ext/src/new_project.rs

midenc-compile/src/stages/parse.rs

bitwalker · 2023-12-04T20:12:02Z

midenc-session/src/lib.rs

@@ -285,13 +285,17 @@ impl Session {
        if self.should_emit(output_type) {
            match self.output_files.path(output_type) {
                OutputFile::Real(path) => {
-                    let path = if let Some(name) = item.name() {
-                        path.with_file_name(name.as_str())
+                    let file_path = if path.is_dir() {


The output file path is guaranteed to always be a file path, not a directory, so this check is unnecessary.

In my case, it's a folder since I'm passing a target folder as output_dir in Session.

Hmm, the OutputFile API is designed to always return a file path, so if it's returning a directory there's a bug somewhere. I'll take a closer look

…anup;

greenhat · 2023-12-05T15:00:17Z

@bitwalker Thank you for the review! I addressed all your comments. I converted this PR to draft and I'm working on it. I outlined for myself what is left to do in the PR description.

and pick up the artifacts from the cargo build to compile the MASM. Similar to the way cargo-component does it. Keep in mind that this cargo extension is planned to be the cargo-component replacement when it comes to building/publishing/etc the Wasm/Miden components. This should explain some implementation details. The overaching idea is to introduce specific to Miden subcommands (e.g. `new`) and pass the rest of the subcommands to cargo where its (e.g. `build`).

…onal

greenhat · 2023-12-07T16:14:01Z

@bitwalker I finished implementing all the review notes and it's ready for review. Please check the updated PR description for a summary.

bitwalker

LGTM. I'll need to look into the OutputFile thing I mentioned, but I'm not going to hold this PR up for that

greenhat commented Nov 10, 2023

View reviewed changes

midenc-driver/src/commands/compile/stages/rewrite.rs Outdated Show resolved Hide resolved

greenhat commented Nov 10, 2023

View reviewed changes

cargo-ext/src/compile.rs Outdated Show resolved Hide resolved

greenhat marked this pull request as ready for review November 15, 2023 12:19

greenhat requested a review from bitwalker November 15, 2023 12:30

greenhat force-pushed the greenhat/ssa-missing-def-i49 branch 2 times, most recently from 46d0dbc to 0727932 Compare November 16, 2023 09:37

greenhat marked this pull request as draft November 17, 2023 14:33

greenhat linked an issue Nov 20, 2023 that may be closed by this pull request

Cargo extension for compiling Rust code to MASM #60

Closed

greenhat changed the base branch from greenhat/ssa-missing-def-i49 to main November 20, 2023 10:52

greenhat force-pushed the greenhat/cargo-ext-i60 branch from 38853fa to 6851153 Compare November 20, 2023 13:56

greenhat added 13 commits November 21, 2023 12:33

chore: draft compile command options using clap

367bb3c

test: draft compile command using the midenc_driver, add test

f09a893

compiling a template project

test: get target dir from CARGO_TARGET_DIR env var when locating Wasm…

e706024

… artifact

fix: emit Session artifact in CodegenStage, don't require Session.mat…

66a3d32

…ches in ApplyRewritesStage Otherwise, ApplyRewritesStage crashes when arg matches are not provided in Session.

test: check target folder exists after build, not before

134fce3

chore: print the progress of the compilation

2db01ff

refactor: make cargo_ext:compile return anyhow::Result

555051b

refactor: use cargo-metadata to get Wasm artifacts

5ad3645

chore: add README.md to cargo-ext

e108242

feat: add new command that generates a new Miden project from a tem…

58d3309

…plate

fix: fix build after the rebase

824a8eb

fix: switch to https://github.com/0xPolygonMiden/rust-templates

62adf34

fix build after rebase

af32c71

greenhat force-pushed the greenhat/cargo-ext-i60 branch from 7f98ec9 to af32c71 Compare November 21, 2023 10:42

test: in cargo extension test create output folder if not exist

91a8b3c

greenhat marked this pull request as ready for review November 21, 2023 15:16

bitwalker requested changes Dec 4, 2023

View reviewed changes

greenhat added 4 commits December 5, 2023 12:09

refactor: move cargo-ext to tools/cargo-miden

29ffb83

chore: remove miden-cargo-extension from workspace deps

426096c

remove next_display_order option in Command

5f0892f

fix: rename compile cargo extension command to build; imports cle…

a69711b

…anup;

greenhat marked this pull request as draft December 5, 2023 13:58

refactor: remove path-absolutize dependency

253d491

greenhat mentioned this pull request Dec 6, 2023

Code style adherence check on CI #78

Closed

greenhat added 2 commits December 7, 2023 15:44

refactor: make WasmTranslationConfig::module_name_fallback non-opti…

769d983

…onal

greenhat marked this pull request as ready for review December 7, 2023 16:08

bitwalker approved these changes Dec 12, 2023

View reviewed changes

bitwalker merged commit 1732c01 into main Dec 12, 2023
1 check passed

bitwalker deleted the greenhat/cargo-ext-i60 branch December 12, 2023 06:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cargo extension for compiling Rust to MASM #61

Cargo extension for compiling Rust to MASM #61

greenhat commented Nov 8, 2023 •

edited

Loading

greenhat commented Nov 15, 2023

bobbinth commented Nov 16, 2023

bitwalker commented Nov 17, 2023

greenhat commented Nov 17, 2023

bobbinth commented Nov 17, 2023

greenhat commented Nov 21, 2023

bitwalker left a comment

bitwalker Dec 4, 2023

bitwalker Dec 4, 2023

greenhat Dec 5, 2023 •

edited

Loading

bitwalker Dec 4, 2023

greenhat Dec 5, 2023

bitwalker Dec 5, 2023 •

edited

Loading

greenhat commented Dec 5, 2023

greenhat commented Dec 7, 2023

bitwalker left a comment

Cargo extension for compiling Rust to MASM #61

Cargo extension for compiling Rust to MASM #61

Conversation

greenhat commented Nov 8, 2023 • edited Loading

greenhat commented Nov 15, 2023

bobbinth commented Nov 16, 2023

bitwalker commented Nov 17, 2023

greenhat commented Nov 17, 2023

bobbinth commented Nov 17, 2023

greenhat commented Nov 21, 2023

bitwalker left a comment

Choose a reason for hiding this comment

bitwalker Dec 4, 2023

Choose a reason for hiding this comment

bitwalker Dec 4, 2023

Choose a reason for hiding this comment

greenhat Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

bitwalker Dec 4, 2023

Choose a reason for hiding this comment

greenhat Dec 5, 2023

Choose a reason for hiding this comment

bitwalker Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

greenhat commented Dec 5, 2023

greenhat commented Dec 7, 2023

bitwalker left a comment

Choose a reason for hiding this comment

greenhat commented Nov 8, 2023 •

edited

Loading

greenhat Dec 5, 2023 •

edited

Loading

bitwalker Dec 5, 2023 •

edited

Loading