Proposal: add JavaScript support via OXC

[shahryarjb]

Requirement

One feature that could significantly enhance Igniter's support for all basic requirements in Phoenix is a robust JavaScript parser or manipulator.
Developing this from scratch would be a large-scale project, requiring considerable time and effort.

Therefore, I suggest considering the Oxc project, which is implemented in Rust, as a potential solution.

---

Where we can use it:

Imagine you want to create a Phoenix component that requires a phx-hook. You would need to modify the app.js file to inject the entire JavaScript file into the Hook object, or alternatively, create a separate JavaScript file, import it into app.js, and then inject it into the hook.

Currently, all of this is done manually in Elixir projects, which involves a lot of documentation and a complex process. This approach is not ideal and can discourage users from using it.

For example, take a look at this object that I injected using the JavaScript spread operator.

hooks: { ...Hooks, CopyMixInstallationHook }

No I want to add GetUserText object.

# This code is from ChatGPT, so it is not totally valid code
let ast = parser.parse().expect("Failed to parse the JS code");
for node in ast.body {
    if let Some(object_literal) = node.as_object_literal() {
        for property in object_literal.properties {
            if property.key == "hooks" {
                property.value.push("GetUserText");
            }
        }
    }
}

Thank you in advance

Refs

• https://discord.com/channels/711271361523351632/1253324141491261472/1297914761438302239
• https://github.com/oxc-project/oxc
• https://oxc.rs/
• https://crates.io/crates/oxc_parser

[TwistingTwists]

https://github.com/TwistingTwists/igniter_js

[shahryarjb]

https://github.com/TwistingTwists/igniter_js

I think if it comes into ash-project org and collaborate, would be good; or even on igniter as core optional installing

[zachdaniel]

Agreed :) Still in the early stages of discovery.

[jimsynz]

I'm happy to work on a Rustler integration if needed.

[zachdaniel]

There is some discussion in the igniter discord channel on the rustler side of things. Would be a wonderful to have more eyes and hands on it

[TwistingTwists]

@jimsynz : Currently I am expanding on ideas on the best way forward to integrate js parser and ast manipulator from elixir.

Here is one approach I have decided to take (unless there are unknown unknowns):
the oxc crate has oxc_parser for parsing js to ast and oxc_ast for ast manipulation.
If we can bind to oxc_ast from elixir, we can get a well maintained + production ready js ast manipulator.

I imagine writing one JS AST manipulator in elixir would be possible, but keeping it upto date would be a hassle.

---

So, the plan is to have ast in rust. Reference that AST struct from elixir. manipulate that ast in js using oxc_ast and give the bindings from elixir.

---

This is the 4 step plan (rough direction) as suggested by @zachdaniel

1. Create a custom Rewrite.Source for js that behaves like the Ex Rewrite source behaves.

2. Create a Zipper data structure for traversing and manipulating AST (this is complicated, I barely understand the Sourceror.Zipper 😆 )

3. Start the work of creating a library of functions to manipulate the zipper, which would be namespaced under something like Igniter.Js.Code.

4. figure out how we can make this available w/o requiring a dependency on rust. There are tools for doing prebuilt binaries, or we'd make it an optional thing that users have to opt into if they want to get js patches in addition to elixir patches.

[TwistingTwists]

Here is what I found about integration rust Struct and elixir

1. create a resource : basically, use Arc<Mutex>

2. expose an nif to take the reference to that resource and do something.

[TwistingTwists]

Here is what I found about integration rust Struct and elixir

1. create a resource : basically, use Arc<Mutex>
2. expose an nif to take the reference to that resource and do something.

same pattern found here as well:

1. create struct using Mutex. Hide it behind Arc
2. Bonus - you can even nest the references to struct if you do Arc<Mutex>

[TwistingTwists]

Here are more code references:

1. create a new resource and return that
2. take resource in 1 as ref do some ops on it.

[zachdaniel]

What is the general solution to #4 above? If we're going to invest a lot of time and energy into this via rustler, we need to have confidence that we can ship this to effectively any platform someone is developing on. The nice thing about pure elixir solutions is that we know they have elixir installed at a minimum 🤣

[TwistingTwists]

What is the general solution to #4 above? If we're going to invest a lot of time and energy into this via rustler, we need to have confidence that we can ship this to effectively any platform someone is developing on. The nice thing about pure elixir solutions is that we know they have elixir installed at a minimum 🤣

This seems to be the easier part.

With RustlerPrecompiled library, this is breeze.

Code change: https://github.com/TwistingTwists/crucible/blob/d686d940aa71c7213dd0ef2ec1b5af0da126988b/lib/native.ex#L6-L11

So, this is already done!! See assets which are precompiled binaries: https://github.com/TwistingTwists/crucible/actions/runs/11458278738
cc @zachdaniel

[TwistingTwists]

What is the general solution to #4 above? If we're going to invest a lot of time and energy into this via rustler, we need to have confidence that we can ship this to effectively any platform someone is developing on. The nice thing about pure elixir solutions is that we know they have elixir installed at a minimum 🤣

This seems to be the easier part.

With RustlerPrecompiled library, this is breeze.

Code change: https://github.com/TwistingTwists/crucible/blob/d686d940aa71c7213dd0ef2ec1b5af0da126988b/lib/native.ex#L6-L11

So, this is already done!! See assets which are precompiled binaries: https://github.com/TwistingTwists/crucible/actions/runs/11458278738 cc @zachdaniel

haven't added support for windows yet. or musl binaries. But this can be done. This is just the barebones apple and linux_x86 support for precompiled binaries.

[zachdaniel]

Windows support will be a hard requirement FWIW

[zachdaniel]

If we want igniter to be truly ubiquitous across the ecosystem, it has to run everywhere elixir runs.

[TwistingTwists]

There is a good news. Can compile for windows, apple and linux (x86 and arm!) woot! 🥳

Here: https://github.com/TwistingTwists/crucible/releases/tag/v0.1.1

[TwistingTwists]

I've had a long holiday in context of diwali in India. Back now.

First update:

Since AST is in Rust, and the AST manipulation is also available via rust methods, my idea is to keep all the AST and AST manipulation in Rust-land.

Here is an API to begin with.

Given a js file path
    - read the js file
    - parse the contents -> return the pointer to AST to Beam
    - write bindings to the manipulation functions so that one can do ____ in existing js file
      - import a new file
      - check if a dependency is already imported

Let's see how this goes.

[shahryarjb]

Since AST is in Rust, and the AST manipulation is also available via rust methods, my idea is to keep all the AST and AST manipulation in Rust-land.

Hi @TwistingTwists
I have a question, if we keep it inside rust how can we validate in elixir? and based on the ast inject a code inside for example app.js hook object

[TwistingTwists]

Since AST is in Rust, and the AST manipulation is also available via rust methods, my idea is to keep all the AST and AST manipulation in Rust-land.

Hi @TwistingTwists I have a question, if we keep it inside rust how can we validate in elixir? and based on the ast inject a code inside for example app.js hook object

The methods for manipulating ast will be present in Rust (oxc_semantic) already does a lot of ast manipulation.

the library will call those methods from elixir (via rustler)

Proposed API on elixir side would be (roughly):

---

For Handling Hooks in app.js

filename = app.js
Crucible.Variable.is_defined("Hooks", filename, "const")
or Crucible.Variable.is_defined("Hooks", filename, "let")

Crucible.Variable.define_if_not_already("const Hooks = {}", filename)

Crucible.Variable.add_kv_in_existing_object("Hooks", filename, key, value)
Crucible.Variable.force_add_kv_in_existing_object("Hooks", filename, key, value)

[shahryarjb]

Hello Dear @zachdaniel,
I think you’re back 🙌🏻🥳! I wanted to ask if this issue is currently a priority. I noticed that recently @TwistingTwists has made some updates to the test repo he created.

Thank you!

[zachdaniel]

It depends on what you mean by priority :) I will not be personally working on it any time soon, but will happily support it and may eventually work on it myself if nothing ends up getting over the line :)

[shahryarjb]

Greetings, friends.

Over the past two weeks, I’ve been learning how to test libraries in Rust to seamlessly integrate them with Igniter and cover all file modification requirements.

Before I explain: All of this is based on my limited knowledge of Rust, so there might be better solutions.

Two approaches came to mind for working with the OXC library:

1. Convert the entire AST to JSON, modify it in Elixir, and send it back to OXC for re-conversion into its native AST format.

Unfortunately, based on this discussion(oxc-project/oxc#7801), deserialization isn’t supported yet and won’t be available anytime soon. This makes this approach completely infeasible. Having this capability would have greatly helped in achieving full access.

2. Write limited features.
   Since creating a custom wrapper is highly resource-intensive and requires a larger team, the scope can be narrowed.
   For instance, I wrote:

• A function to add an import.
• A function to remove an import.
• A function to add an object name to a hook.

In this case, the AST doesn’t get imported into Elixir. Instead, specific requirements are met by writing tailored functions for each need.

---

I’ve already explained about OXC above. I hope you can give me some advice. Right now, I can fulfill the needs of my own library, but my goal is to establish a more foundational approach and also integrate it with Igniter. I’d really appreciate hearing your thoughts.

The OXC library is actually inspired by another project called BiomeJS (https://github.com/biomejs/biome/tree/main/crates), but it has taken a completely different direction. The BiomeJS project is very active, and it seems to support deserialization. However, working with it would require at least two weeks of effort before making a decision. You can check the link above to see its submodules.

I’ve worked a bit with this library, but its documentation didn’t meet my requirements, so I couldn’t make much progress. It seems like it would take around one to two weeks to properly evaluate it.

What are your thoughts and suggestions?

It's worth mentioning that all the libraries in question, whether for Rust or Zig, are primarily designed for JavaScript developers. Unfortunately, they often lack proper documentation that could serve as a guide for learning and using Rust. As a result, you usually have to rely on function names, some comments, or the type system to figure things out, often through trial and error.

Limited functions like these

Refs:

• Deserializing Serde JSON AST to Program oxc-project/oxc#7801
• https://github.com/biomejs/biome/tree/main/crates
• https://discord.com/channels/711271361523351632/1253324141491261472/1316398710330097714

Update

For Biome

• How to Convert JSON Back to AST Tree in Rust? biomejs/biome#4727

[zachdaniel]

I think we really only have two options based on all of this:

1. have rust tools that do very specific things, in rust specifically, just returning to us a diff to make to a file.
2. write parsers in Elixir and just do it ourselves.

Obviously #2 is a huge undertaking. So we probably start with #1 and then maybe someday tackle #2.

I would like to make igniter_js a separate dev package that gets installed by igniter scripts that need it also I think, so we can iterate on it independently and try different things.

[shahryarjb]

Hi @zachdaniel , I hope you're doing well.

The approach you mentioned for porting tools to Rust seems logical and reasonable. However, my primary concern revolves around using JavaScript parser projects.

Let me clarify with an example: imagine building an AST in Rust for JavaScript. Each part of a node, such as a function or a variable, has a specific type (like a struct) that allows for seamless operations within Rust.

Now, let’s assume we want to use this in Elixir. The first issue is that we’d need to rewrite all the types in Elixir. This essentially means recreating the entire JavaScript parser from scratch, defeating the purpose of using Rust.

If we convert it to JSON, it can be easily consumed in Elixir. However, the problem arises when we want to modify that JSON in Elixir and send it back to Rust to be converted into an AST with its original types. This is the core issue: in Elixir, we can’t make AST-like modifications as we would with macros.

---

So, what are the potential solutions?

1. Write deserialization logic for everything in the mentioned library within Rust, which is impractical.
2. Write a wrapper, which can be very lengthy and cumbersome.
3. Limit the requirements—this might solve my specific use case, like My UI library, but it forces anyone with different needs to implement their solution in Rust, leaving no general-purpose tool for Elixir.

---

This issue is particularly evident with parsers because they essentially create constructs that don’t natively exist in Rust. They involve analyzing and managing numerous types, lifetimes, and mechanisms to prevent panics—all of which don’t align well with Elixir.

If we were to adapt it for Elixir’s format, we’d also need to create a reverse transformation format to convert data from Elixir back into Rust.

For example, I haven’t found a way to handle allocators in Elixir, so I’m forced to manage them directly in Rust.

This is the problem I was trying to highlight. I don’t have opinions on other features; my focus has been specifically on parsers. My observations are based on examining these two projects and discussing them with their maintainers.

That said, your suggestions make perfect sense. It’s entirely feasible to use tools that don’t have the same challenges as parsers. For instance, tools for CLI utilities could be adapted from Rust’s ecosystem to Elixir without major issues.

Regarding writing a full JavaScript parser in Elixir, I don’t have any experience. I believe it would require a dedicated team and potentially months, if not years, of effort.

Apologies if my translated explanation didn’t fully capture the nuances of my perspective. I tried my best to outline the challenges I’ve encountered.

Best,
Shahryar

[zachdaniel]

I'm following the issue, and what it seems like is that it's going to be massive amounts of work no matter how we cut it to have js patching in Elixir in any kind of fluid way. So I think we just make very specific utilities, like you mentioned, and just leave all the rest for later for now. We can tell people "the way we patch js is by writing rust, sorry 🤷".

Later on down the road we can figure out ways to make it nicer. Rust isn't all that bad 😋

[zachdaniel]

We can provide a basic utility to say something like "patch this text" which passes text to a rust script which passes us the new text back, so that it still sits nicely in the igniter framework.

[shahryarjb]

@zachdaniel
I think your points make sense to me. I believe we can start with a few cases that I’ve already tested, as shown in the image above, and gradually expand on them. We can create a lightweight package for common tasks to get started.

For example:

• Is this module imported?
• Does a hook exist?
• If a hook exists, is our target object attached to it?
• Which objects should we remove from the hook?

For now, these examples are sufficient for my own library for example. The user can write their JavaScript in a separate file and make changes there, rather than directly modifying app.js.

[zachdaniel]

Yep, basic fit-for-purpose tools sounds perfect to me as a start, and we can continue to explore the space as necessary <3