Rust contributions?

[loafofpiecrust]

I want to contribute to Nix and the surrounding ecosystem, but I find that modifying existing code or writing new code in C++ is much more error-prone for me and I (along with others, I'm sure) could give much more meaningful contributions in Rust.

I've noticed overlap in the Rust and Nix communities and I'm sure that having a path to contribution here in Rust would energize people and Rust anyway fits really well with the Nix idea of static guarantees about a system.

I see that there's a nix-rust folder where someone attempted to port a piece of Nix to rust, but this project was abandoned. It seems like there's solid C++-Rust interop now with the cxx project, and I might be able to craft an initial PR hooking it up.

Would maintainers be willing to support a PR of this nature? Further, would the project be open to future PRs in Rust for new features or rewriting existing code if it's significantly simpler?

[nrdxp]

FWIW, I would love Rust interop with the Nix C++ codebase. I am very much in your boat and find my time in Rust much more productive. There may even been cases where Rust could make our lives a lot easier.

In the longer term, a Rust API to the Nix daemon would also be really interesting 🤔

[thufschmitt]

I might be able to craft an initial PR hooking it up.

Would maintainers be willing to support a PR of this nature? Further, would the project be open to future PRs in Rust for new > features or rewriting existing code if it's significantly simpler?

Not speaking in anyone’s name except my own (esp. given that I wasn’t involved in the previous Rust experiment), but I think the main reason why the previous experiment got abandoned was that it was adding a lot of complexity while not providing a lot of benefits (IIRC the only part that was written in rust was the StorePath abstraction, which was arguably much nicer in rust, but was also pretty small, so not really worth all the trouble).

So imho, Rust contributions could be totally acceptable (and even great, I for one would be happy to be able to write Rust code rather than c++), provided that

1. The result is easier to work with than the old nix-rust setup (in terms of build-system in particular, I think I remember that was a bit of a pain)
2. There’s some real and consistent benefit from it (maybe some kind of plan on how to migrate a substantial part of the codebase to Rust as that’ll probably not be something that will happen overnight nor spontaneously)

[fogti]

I'd like to collect previous efforts on the nix-rust partial-port side (the further down the list the more it deviates from nix):

• my old PR: nix-rust: various optimizations #4697, improvements to the old nix-rust code base
• https://github.com/griff/Nix.rs, some parts of nix reimplemented in rust
• https://git.ytrizja.de/zseri/yzix, based on mostly the same concepts as nix (but it's not really a reimplementation), but uses a different store layout, different interface, and tries to also integrate ninja, ccache and distcc, basically a much more researchy project, and less useful; but it might be possible to extract common abstractions out into some shared rust crates.
• https://www.reddit.com/r/NixOS/comments/r6ykln/tvix_we_are_rewriting_nix/hmxk04p/ TVL / tvix tries to reimplement the Nix evaluator in Rust.

cc @griff

[ldesgoui]

Hey, I'm also trying to start something (https://github.com/ldesgoui/rust-in-nix), I wanted to prepare a little intro to give context, but I don't have the time immediately, the gist of what I'd like to achieve is in the README.
Happy to get in touch over Matrix

[fogti]

As it appears that there are now at least 4 tries to reimplement parts of Nix in Rust, it might be appropriate to isolate common functionality or patterns into crates that can be shared between all of them (as it might be likely that we run into edge cases which could be modeled with a big variety). It is also easy to see that the initial development targets of the associated projects vary to a somewhat greater degree. Maybe I should create a GitHub org to organize and bundle these efforts. (especially because I don't want to litter this issue thread with meta-talk)

edit: created https://github.com/RIIR-Nix

[fogti]

@ldesgoui

Happy to get in touch over Matrix

what's your Matrix identifier (username and server)? (mine's @zseri:matrix.org)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/the-uncompromising-nix-code-formatter/17385/39

[kamadorueda]

I noticed that Flex and GNU bison with %glr-parser support as required by Nix ambiguous grammar, used in the Nix lexer and parser, do not have an equivalent in rust. There is lalrpop, but that is.. LALR, and Nix requires GLR or Earley (since it is an ambiguous grammar)

So I just created a library for it, and used it to port the lexing and parsing components to Rust (source here)

From this to having a Nix evaluator written in Rust we are very very close: just visit each node and perform the small set of actions in parser.y to build the symbol table, etc

The result is easier to work with than the old nix-rust setup (in terms of build-system in particular, I think I remember that was a bit of a pain)

• The rust port looks readable, someone familiar with Nix's lexer.y and parser.y will feel at home
• Since it is rust and only rust it is also easy and comfortable to work with (cargo, rust-analyzer, and all of that modern tooling niceties)

There’s some real and consistent benefit from it (maybe some kind of plan on how to migrate a substantial part of the codebase to Rust as that’ll probably not be something that will happen overnight nor spontaneously)

I suppose the advantages derive from this being written in rust:

• Fearless concurrency in the lexing and parsing stages
• Possibility to cache the lexing and parsing phases, since those are immutable on the input
• Possibility to create a yet-to-evaluate graph and perform as much evaluations in parallel as the imports graph allows at any given time
• Avoid memory leaks, which existing c++ code mentions here and there

---

Since we would be using something based on Flex and GNU Bison, we reduce the possible bugs due to the simple fact of changing the lexing and parsing engine, and worst, the bugs that would be created by having to change the grammar, actions, or lexing states to accommodate ourselves to something else than flex-bison

---

A small demo:

echo 'let a = 123; in a'| cargo run --example nix

Lexemes:
  LET "let" (1, 1)
  ID "a" (1, 5)
  = "=" (1, 7)
  INT "123" (1, 9)
  ; ";" (1, 12)
  IN "in" (1, 14)
  ID "a" (1, 17)

AST:
Γ
  expr
    expr_function
      LET "let" (1, 1)
      binds
        binds
        attrpath
          attr
            ID "a" (1, 5)
        = "=" (1, 7)
        expr
          expr_function
            expr_if
              expr_op
                expr_app
                  expr_select
                    expr_simple
                      INT "123" (1, 9)
        ; ";" (1, 12)
      IN "in" (1, 14)
      expr_function
        expr_if
          expr_op
            expr_app
              expr_select
                expr_simple
                  ID "a" (1, 17)

Santiago is still a work in progress, for instance it is missing Bison's %left %right and %nonassoc statements, and instead it returns all possible ASTs according to the ambiguous grammar, but that should be simple to implement, and anyway is a very relevant library to this issue

[kamadorueda]

I got the Nix AST in rust: https://github.com/kamadorueda/nixel

From this point, if we wanted to have a Nix evaluator in Rust we would need to add the EvalState abstraction, build the symbols table, the built-ins and the "Thunk" abstraction to evaluate stuff lazily

[fogti]

The imo hardest builtin to correctly implement/"find" is builtins.match, because it uses the C++ regex engine, and it is not simple to replicate it because every regex engine seems to use slightly different syntax. It's doable, sure, but probably takes some effort to port or wrap (as in: providing a simple rust wrapper which basically links against the snippet...)

[kamadorueda]

@zseri One step at a time

[fogti]

Just mentioned it because besides YZITE/nix2js#2, in nix2js that was an important problem, and the standard rust regex crate does afaik use a different syntax than the one expected by builtins.match.

[kamadorueda]

Okay, so if someone wonders what a Nix evaluator in Rust would actually look like, this would be a good example:
https://github.com/kamadorueda/toros

Currently able to evaluate things like:

let
  add = a: b: a + b;
in
  add 1 2

Which involves complex things like scopes, built-ins, currying, identifiers and laziness

Probably the tests folder is a good place to see the current capabilities

I also saw a few interesting possibilities, like compiling to Web-Assembly, embedding into Jupyter notebooks, and defining the store interface as a trait so that people can bring their own amazing back-end (s3? ipfs? dna?)

Rename Rc to Arc and boom, parallel evaluation in the corner. Hash of input file and boom, file-level cache. And so on

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-09-nix-team-meeting-minutes-15/23951/1

[nyabinary]

Any updates on this?

[Ericson2314]

See what the Tvix people are up to.

[nyabinary]

@Ericson2314 I thought Tvix wasn't meant as a replacement to Nix, is there any chance in the future that people will be able to contribute to Nix in Rust?

[nrdxp]

@nyabinary, the stated goal in the material I read, at least, was to create an Nix evaluator that was capable, eventually of course, of evaluating all of nixpkgs, so in that sense it should definitely be usable as a full replacement (once it's ready).

[nyabinary]

@nrdxp Would a gradual replacement be possible maybe? Also, a Matrix room for this effort would be really nice to have.

[flokli]

@nyabinary if you want to stay up to date with Tvix, you can join the IRC channel through Matrix.

[Ericson2314]

A gradual replacement is very possible. I have done lots of work to make the daemon protocol be better documented. Now, I really hope tvix will implement the daemon protocol. Then we can mix and match stores/evaluators from either project.

[nyabinary]

A gradual replacement is very possible. I have done lots of work to make the daemon protocol be better documented. Now, I really hope tvix will implement the daemon protocol. Then we can mix and match stores/evaluators from either project.

That would be absolutely great, considering tvix does some very smart things in their reimplementation.

[Ericson2314]

Exactly!