-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow composing overrides #4004
base: master
Are you sure you want to change the base?
Conversation
It's not entirely clear to me what this means since flake inputs can already have overrides. What is a higher level override? |
Flakes can have overrides, but as far as I can tell overrides for flake inputs are not currently working. A trivial example of A -> B (override C.D) -> C -> D doesn't currently apply the override. Overriding with higher level overrides means using A's override rather than B's in an example like this: Guess I should update the commit to explain that more clearly? |
I'm not sure I understand the notation "A -> B (override C.D) -> C -> D". What does that mean exactly? |
Sorry flake A has input flake B etc. Like this: https://gist.github.com/mkenigs/7c07c53bfab2825fa9f9b03adcd59b79 |
Does that make sense now/seem like an improvement? |
@edolstra anything I can cleanup or explain for this? Or should I close it? |
No, please keep it open, I'll try to look at it soon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, the gist motivates it well since I would expect that overrides composed analogously to overlays.
51cb2d4
to
8ec1703
Compare
Yeah, flakes are a great place where you'd want multiple inheritance, as implemented in pop.nix from #114449 |
Are you reinventing multiple-inheritance, but without confronting the fact openly? |
Not that I'm aware of, I think I'd need you to explain that a little more though. I'm just viewing flakes as functions, but the top level flake can decide the inputs of lower level functions. So if I define a bunch of flakes: |
I just ran into this problem when having three levels of flakes A -> B -> C where I want C to follow an input from B and not the input from A that has the same name. The current behaviour is pretty confusing, so I'd love to see this merged! assuming follows uses the same implementation as overrides: #3602 (comment) |
I just ran into this problem with three levels of flakes, too. Mine was where the third flake followed one of its own inputs. It ended up trying to resolve with the second flake as the root instead of the first, which was actually fine for building the second flake (b) but broke when I moved onto the first, outer flake (a). It resulted in a "follows a nonexistent input 'b/c/some/thing'" error. It was supposed to look at 'a/b/c/some/thing'. I reported it here and might merge it with this bug but can't quite tell if it's the same exact bug because they talk about circular refs and stuff and you don't need to introduce circular refs to trigger this one. |
I marked this as stale due to inactivity. → More info |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Idea looks good, the tests describe the behavior that we want from Nix. I didn't look very closely at the C++ side since it's probably a bit outdated at the moment.
tests/flakes.sh
Outdated
|
||
# B, C, and C2 same as previous test | ||
# A + B + C = 3 | ||
[[ $(nix eval $flakeA#foo --recreate-lock-file) = 3 ]] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this --recreate-lock-file
be necessary? Ideally Nix detects that the override in flakeA has been removed and updates the lock file accordingly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dropped a bunch of them but the rest are necessary for tests to pass. I haven't looked into why
Converted to draft because pending changes from review. |
Thanks for the review @edolstra! I'll rebase this when I get a chance, but it might take me a while to find time to brush up on the codepath it's changing |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2023-06-12-nix-team-meeting-minutes-62/29315/1 |
Does this also fix #8325 ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
<3 Thanks @mkenigs
I ran out of steam writing more suggestions, but you get the idea.
This allows treating "nodes" in the override graph as empty if they don't have a FlakeRef, which allows combining overrides set by different flakes
Allow multi-level overridees, overriding with higher level overrides when appropriate. See tests for the following motivating cases: - multi-level overrides: A(override B.C.D=D2) -> B -> C -> D - overriding an override: A (override B.C.D) -> B (override C.D)-> C -> D - overrides are merged: A (override B.C.D.E) -> B (override C.D)-> C -> D -> E The prior implementation used a single overrides data structure accessible throughout recursive calls to computeLocks. Instead, only pass each recursive call exactly the overrides it needs. This better protects individual recursive calls from modifying the overrides "global" to the recursion.
tests/flakes/follow-paths.sh
Outdated
@@ -146,5 +146,5 @@ EOF | |||
|
|||
git -C $flakeFollowsA add flake.nix | |||
|
|||
nix flake lock $flakeFollowsA 2>&1 | grep "warning: input 'B' has an override for a non-existent input 'invalid'" | |||
nix flake lock $flakeFollowsA 2>&1 | grep "warning: input 'B' has a follows for a non-existent input 'invalid'" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if it's helpful to have this distinction or not - can revert if not
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if we should make this distinction but if we do want to keep it this way I suggest changing the message to input 'B' follows a non-existent input
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
Pushed a bunch of quotes but let me know if I missed any |
I think it may - that's part of the reason for 973eb60. Unless there's a difference between follows and overrides, that's tested in this test case https://github.com/NixOS/nix/pull/4004/files#diff-4682489d70e9a85daa7f2a8a11cc25dd59a602399cd6e0b96b52eaa610e2cf8bR119 |
Updated all the C++ and is ready for re-review! |
@edolstra @Ericson2314 is there still any interest in this? |
I am happy to resolve merge conflicts if anyone ever wants to take a look at this |
Allow multi-level overridees, overriding with higher level overrides
when appropriate.
See tests for the following motivating cases:
The prior implementation used a single overrides data structure
accessible throughout recursive calls to computeLocks. Instead, only
pass each recursive call exactly the overrides it needs. This better
protects individual recursive calls from modifying the overrides
"global" to the recursion.
This change depends on allowing no FlakeRef for FlakeInput.
That allows treating "nodes" in the override graph as empty if they
don't have a FlakeRef, which allows combining overrides set by different
flakes