Nickel OS / nixpkgs parsing / Nickel as a full replacement for Nix

[crabdancing]

Has there been any discussion of Nickel replacing nix as a backward-compatible parser for large repos like nixpkgs, thus allowing it to become a more correctness-focused package manager alternative to Nix. I doubt we're there yet, but has any roadmap been discussed, or is this out of scope of Nickel for the moment?

[yannham]

Hello,

Some work has been started toward this direction a long time ago, in #1067. The PR was stalled because the original author left and we had other priorities back then. His focus was to transpile Nix code (parse and convert to Nickel), which isn't insanely complicated for most of the Nix language, which is rather simple as a language. There are some corner cases of the semantics and subtelties around environments because of Nix with, and inherit was missing as well, but still doable.

However, this doesn't handle any Nix primop, such as derivation. For those, one possibility is just to call to the Nix C++ API under the hood, which is in fact already possible currently - if you build nickel with the experimental-nix feature, there is an %eval_nix% primop which takes a string representing a Nix expression and forward that to Nix evaluator, and then serialize the result back. It's messy and effectful, but it has the merit of being there. Note that this is for primops that have an actual side-effect, like populating the store. I suspect many primops must be implemented in pure Nickel (the one manipulating attribute sets or string contexts for example) as a compatibility layer.

Besides technicalities, one of our problem so far is that using Nickel just as an alternative Nix VM doesn't really leverage Nickel's strong points. Sure, there is some stuff we can make better, but in an ideal world a hypothetical Nickelpkgs would be architectured in a totally different way than Nixpkgs. Maintaining backward compatibility with Nixpkgs is a big weight.

If the goal is to use Nickel for your own config, and to leverage Nixpkgs for the existing stuff but without adding Nickel code inside Nixpkgs, then I guess the constraints are more manageable. Organist is able to pull stuff from Nixpkgs from "within Nickel" (it's not entirely true, but it looks like it) by serializing data as JSON and having both a Nix lib and a Nickel lib managing the interop between the two worlds. It's far from being as powerful as being able to call arbitrary Nix from Nickel, but it's enough for the purpose of taking some packages out from Nixpkgs to put in a developer environment, keeping the Nickel part quite idiomatic.

In general, there's been quite a lot of conversation around this in the RFC003 PR, that has been stalling for a while. I must also mention some contributor on the discord named Austin (it might be @ajbt200128 ? but I'm not sure - sorry for the unsolicited ping if it wasn't) has been working on taking up the work of Nix transpilation, and seemed to have made good progress. But unfortunately no PR has been opened and I don't know what's the current status of this effort.

Maybe another interesting direction would be to explore the usage of Tvix, which is a re-implementation of Nix in Rust. Beside the fact that they could get rid of a lot of technical debts, Nickel being written in Rust as well, re-using entire part of Tvix could be much easier than interfacing with Nix directly or re-implementing everything on the Nickel side.

Summary

• the idea of transpiling Nix to Nickel and running everything there has already been explored in Nix/nickel #1067. The work is unfinished, and stalled for now, but wasn't far from covering all of the "pure part" of Nix.
• beside transpilation, we would need a compatibility layer that re-implements all Nix builtins in Nickel. There is an experimental %eval_nix% in Nickel that could help for effectful primops.
• I anticipate that once you start to test on the whole of Nixpkgs, we will have many things to fix to exactly match unwritten Nix semantics "bug for bug". From what I know, this has been a big effort for Tvix, not to be underestimated.
• If you accept a less featureful interop, you can probably get by using JSON as an interop format and a bit of butter on both the Nix side and the Nickel side, as done in organist.

For now this line of work is paused, and we are focused on delivering a bytecode virtual machine in order to fix performance issues. However, after this, I plan to dedicate a big chunk of time to going back to Nickel use-cases, including Nix.

[ajbt200128]

Hey! My branch with the progress I've made is here https://github.com/ajbt200128/nickel/commits/nix/nickel/ I haven't been able to get back to it, but I believe all that was left was fixing some scoping issues with defaults (although they may be fine to leave as is), the inherit keyword (I have some uncommitted work on this that I can clean up and push), and then import (this should be trivial). There's also the matter of binding the nix builtins somehow, but we could just bind those with %eval_nix% or something.

I haven't had too much time to finish these parts, but I'm open to cleaning them up, or hopping on a call with someone to go over what's left. I wanted to wait until we had inherit/import until I opened up a PR, but am open to merging what's there now, but that's up to the maintainers

[yannham]

Thanks for the heads up, @ajbt200128, and thanks for doing that! I think our priority would be to get what you have now merged, even if it's unfinished - we can put that behind an experimental feature flag. Then it's part of the CI and it'll evolve alongside the main code base. It could also be picked up by you later, or someone else. Unmerged stuff, on the other hand, is undiscoverable and anyone trying to pick it up might be put off by the huge code drift if the work is paused for too long. That's partly what happened to the first unfinished Nix transpilation PR.

[ajbt200128]

Yea totally fair! I'll try and clean up what I have and get it ready to merge sometime in the next week or two then!