I Started a project like this, here is what I learned.

[MrVintage710]

Hey Mafii! I know this is a wierd way to send a message, but this is the only way I found as I am not on any socials. My name is Brooks, and I am a Developer from Montana who also worked on a project just like this. Here is the repo for that project: https://github.com/MrVintage710/bevy-yarn-spinner

This was a very fun project to tackle, and I dont want to dissuade you from working on it. But, there are some problems that I ran into later in development that I thought that you may want to think about.

First and foremost, Yarnspinner was designed to fit into the Unity game engine, so when developing certain language features you may have a hard time integrating that for the rust ecosystem. There are features of the language that would require alot of work for the runtime, mainly because you are trying to implement systems from the Unity game engine. Things like defining custom functions or commands. In my crate, I took a macro aproach that is tricky, but definately the most user friendly in rust. Make sure you think about how this will tie into your chosen frame work (Bevy).

Second, there are some problems with the syntax of the language that make it difficult to be able to package with bevy when using wasm builds. This is a minor complaint, but there are some parts of the language that can be very ambiguos unless the correct spacing is given. for example:

---

Companion: Hi there! What do you feel like doing today?

-> Player: I want to go swimming.
Companion: Okay, let's go swimming.
-> Player: I'd prefer to go hiking.
Companion: Cool, we'll go hiking then.

Player: Sounds good!

---

This is an example from their docs. Without the information of the tabs, you wouldn't be able to tell what block that last line is a part of. This means that whitespace is important in the language. If you want to do compression for the language in cases where file size matters (AKA Wasm builds of Bevy) There can be alot of bloat with more dialog heavy games.

Third, Yarn spinner requires alot of static states. This goes against rust's manifesto, however this can be mitigated but the Bevy ECS. This isn't game ending, but it sure does make some features annoying to add in.

Fourth, and probably the most challenging aspect, Yarn Spinner doesn't have a language spec to follow. This is important to have, as even the smallest desitions on your end could make older scripts not work. To make your compiler accurate, there would need to be alot of test to make sure every aspect of the language is the same (ltr evaluation vs rtl, float accuaracy, comparing ints and floats, ect)
This last is what ultimately made me give up on transfering this language to rust.

However, I hope that this project is fruitful! Solve all the problems I couldn't / didn't want to. If you have any questions, leave them here and I will answer them to the best of my ability. Bonne chance!

PS: I didn't quit completely. I still think that nartive tools are important for the bevy community. So I made my own language! It is called Libretto script, and it are working on that project here: https://github.com/MrVintage710/libretto. If that sounds interesting to you, give it a quick glance!

[Mafii]

Thanks so much for the insights! This will surely help me with certain decisions - for now, my goal is to write a compiler for yarn spinner (which will be challenging, as there's a lot of funny nuances with the antlr syntax like different modes, which makes it harder to understand and re-write it without such a thing), and then see what I can do. Good luck with your project!

[janhohenheim]

I'm not @Mafii, but I'll take the liberty to answer this as a collaborator :)

Thanks for sharing your experience, this is super useful!

Things like defining custom functions or commands. In my crate, I took a macro aproach that is tricky, but definately the most user friendly in rust. Make sure you think about how this will tie into your chosen frame work (Bevy).

I was thinking this could be done via one-shot systems or by using a function registering mechanism similar to how I implemented spew.

Without the information of the tabs, you wouldn't be able to tell what block that last line is a part of. This means that whitespace is important in the language. If you want to do compression for the language in cases where file size matters (AKA Wasm builds of Bevy) There can be a lot of bloat with more dialog heavy games.

Huh, I was not aware of this, thanks for pointing it out. I guess the only way to deal with this is to add a disclaimer to the readme?

Third, Yarn spinner requires alot of static states. This goes against rust's manifesto, however this can be mitigated but the Bevy ECS. This isn't game ending, but it sure does make some features annoying to add in.

Yeah, it sounds like pulling this data out of the Bevy World would be easier.

Fourth, and probably the most challenging aspect, Yarn Spinner doesn't have a language spec to follow. This is important to have, as even the smallest desitions on your end could make older scripts not work. To make your compiler accurate, there would need to be a lot of test to make sure every aspect of the language is the same

I think it is okay to break compatibility with current Yarn files, since nearly all are written exclusively for Unity projects and would not ever be compiled using this crate anyways. It should be enough to have tests in place that "document" our current guarantees, whatever they might end up being, and try to not break these.

---

Overall, your comment has shifted my ideas about how to best organize this project. While going compiler -> stand-alone runtime -> Bevy plugin would be nice for the Rust gamedev community as a whole, it might be significantly easier to integrate the runtime directly into the Bevy plugin. Thoughts on this, @Mafii and maybe @TimJentzsch?

[TimJentzsch]

Overall, your comment has shifted my ideas about how to best organize this project. While going compiler -> stand-alone runtime -> Bevy plugin would be nice for the Rust gamedev community as a whole, it might be significantly easier to integrate the runtime directly into the Bevy plugin.

Yea, that could indeed be easier to accomplish.
Given the multitude of potential challenges here, I would suggest to maybe start with a minimal prototype that implements the most challenging features (e.g. passing data around, defining custom functions, etc.) instead of implementing the whole spec and then trying to integrate it into Bevy.
This might help to iterate on the problems and see if they are solvable.

An other alternative to consider would be to ditch yarn spinner and to come up with a solution that is easier to integrate into Rust & Bevy.
Of course that would be an even bigger task and a new project altogether.
There would be many aspects to consider, especially localization can be hard to get right (but is essential in this context).
The project could take learnings and inspiration from Yarn Spinner or other projects.
But it would definitely be a huge undertaking and no simple feat.

[janhohenheim]

@Mafii and me were also talking about keeping Yarn syntax parity, i.e. everything that Yarn Spinner accepts will also be accepted by us, but deliberately not runtime parity, i.e. the runtime is allowed to interpret the Yarn files slightly differently and might make accessing features very different.
This would warrant not calling this project rusty-yarn-spinner, but something else containing the word yarn but not spinner to stress the connections and yet differences to Yarn Spinner. Kinda like how GGRS references GGPO.

[MrVintage710]

I like that! That would allow people to use the language look and feel that they like but it would still work well in the rust ecosystem. And, with enough work, you can add more support for legacy yarnspinner code.

If there is anything you want from my yarn-spinner repo, please use it as I haven't worked on that project since last year.

Can't wait to see where this goes!

[MrVintage710]

@TimJentzsch Ditching yarn spinner is the desition I have made. If you all make that desition (I would say try this implemetation first) I would love to chat with you guys and work together on a solution. As previously stated, my iteration of this naritive tool is libretto, https://github.com/MrVintage710/libretto, however I would love to cooperate on a spec. Reach and let me know if that is something your team would want to do. In the following days I will make a true spec for my scripting language so that it can be critiqued by the community.

[Mafii]

Hello @MrVintage710, today I gave up on manually writing the parser. But it's not as bad as it sounds!

Amazingly, I was able to get an ANTLR generator for rust working, which means this repository now has a feature-matching (and complete) parser for the yarn language - matching 100% with the v2.3.0 tag of YarnSpinner!

@janhohenheim (absolute legend) is porting the compiler/parser tests, so we will have a test suite to track if everything works.

This also means that with little effort (as long as the yarn spinner team does not use a version that is newer than the supported one in antlr4rust) an updated (or outdated) syntax can be supported! I've documented my steps in a markdown file so anyone should be able to do it!

I'm closing this issue, but feel free to respond in here or open new issues for discussion!

[MrVintage710]

This is great! That seems to be a really easy way to do it! 👏

[janhohenheim]

@Mafii

[janhohenheim]

@MrVintage710 I thought you might like this. I managed to get custom functions working with the best possible API: the user simply passes a name and a plain old function: https://github.com/Mafii/rusty-yarn-spinner/blob/1c308f3b176364b0ee81ef74bbdf2c87eed80f65/crates/core/src/yarn_fn/function_registry.rs#L57-L172

The Rust calls to evaluate them and get a return value out are a bit verbose, but that's fine since that part is never user-facing, as they will only register fns in Rust and only call them in Yarn.

[MrVintage710]

This is super clean for functions! Is there auto conversions from Yarnspinner types to rust types? I think that Mafii was right when he said that you were a legend!

[janhohenheim]

Thanks! There are From, TryFrom, Into and TryInto impls for all permutations in place where the original had them. I ended up implementing a subset of dotnet's Convert type to make sure the behavior is more of less the same: https://github.com/Mafii/rusty-yarn-spinner/blob/57d7011c7eb61262b3ace073c9ec813b54d0e22c/crates/core/src/value/convertible.rs