package overrides using a mini module system #312432

Atemu · 2024-05-17T12:01:06Z

Description of changes

Putting a package's configuration options into function arguments in order to expose them via override is a well established pattern but has major issues:

No discoverablilty/documentation of available options
Namespace collisions (calling an option "xorg" would result in callPackage inserting the xorg package set)
Odd syntax and scoping limitations when declaring options (? and no ability to use let bindings)
No type checking
No good error messages
No sane way to transparently communicate breaking changes

But we have a well-established solution to all those problems: The module system.

This is a little thought that came to my mind while thinking about RFC 169:
Why couldn't we just use the module system for configuring packages?

The extent to which this is done here is merely the configuration options that were explicitly exposed by the package before which is similar to how modules are used NixOS; explicit options provided by the package maintainer to customise some specific aspect of the package.

The only limitation I currently see is that it is a bit awkward to compose multiple overrides:

# This is fine
(mlterm.override {
  configuration = {
    gui = {
      wayland = false;
      sdl2 = false;
    };
  };
}).override
  # This requires advanced knowledge of the moule system and overrides
  (prev: {
    configuration = {
      imports = [ prev.configuration ];
      gui.sdl2 = true;
    };
  })

This could however be remedied using a dedicated overrideFeatures function.

As for performance:

Edit: Out of date. See below.

[atemu@HEPHAISTOS nixpkgs]$ hyperfine --warmup 1 -r 30 'nix-instantiate -A ffmpeg'
Benchmark 1: nix-instantiate -A ffmpeg
  Time (mean ± σ):     716.5 ms ±  13.9 ms    [User: 430.6 ms, System: 143.5 ms]
  Range (min … max):   658.7 ms … 738.2 ms    30 runs
  
[atemu@HEPHAISTOS nixpkgs]$ hyperfine --warmup 10 -r 30 'nix-instantiate -A mlterm'
Benchmark 1: nix-instantiate -A mlterm
  Time (mean ± σ):     739.5 ms ±  33.0 ms    [User: 445.2 ms, System: 151.8 ms]
  Range (min … max):   688.9 ms … 792.6 ms    30 runs

With modules:

[atemu@HEPHAISTOS nixpkgs]$ hyperfine --warmup 1 -r 30 'nix-instantiate -A ffmpeg'
Benchmark 1: nix-instantiate -A ffmpeg
  Time (mean ± σ):     712.8 ms ±  21.4 ms    [User: 428.8 ms, System: 143.0 ms]
  Range (min … max):   657.3 ms … 748.0 ms    30 runs
  
[atemu@HEPHAISTOS nixpkgs]$ hyperfine --warmup 10 -r 30 'nix-instantiate -A mlterm'
Benchmark 1: nix-instantiate -A mlterm
  Time (mean ± σ):     740.9 ms ±  35.9 ms    [User: 446.8 ms, System: 152.2 ms]
  Range (min … max):   687.8 ms … 795.2 ms    30 runs

There might be a slight difference here but none that I could measure using runtime.

Things done

Add a 👍 reaction to pull requests you find important.

roberth · 2024-05-17T13:36:12Z

My thoughts on this draft:

Unfortunately we can't measure the performance impact by adding only a handful of examples.
The performance impact of the module system is bounded by how much you ask it to do.
That means that we don't have to expect the impact of, say, loading NixOS's massive module list, but something very small for small configs like these.
However, that means that for a realistic picture, we need to apply this solution to all packages in ffmpeg's closure, or a known portion for which we get a statistically significant difference.

With the Nixpkgs architecture team, we've looked into applying the module system more broadly. See e.g.

Performance benchmark for drv-parts/dream2nix compared to bare derivations #295030

Reducing the scope of module system usage may alleviate the performance issue somewhat, but then the question is whether this is the most effective allocation of our evaluation performance "budget". (I don't have an answer)
By applying the solution to only part of the problem, I feel like we're almost making the situation worse because of the added complexity of having different solutions for what are actually instances of the same problem - package configuration is just parameters, just like dependencies are, or "environment" things like the platforms. Shouldn't they all be discoverable, mergeable, etc?

doronbehar

I like this! Not because I'm hopeful about the performance improvement (in fact I'm skeptic about this detail), but mostly because finally options have a description written with Nix and not in a comment.

AndersonTorres · 2024-06-02T12:15:52Z

Obviouslt my question is: how hard will be to teach this to newcomers?

Atemu · 2024-06-02T15:55:26Z

Teach what to newcomers? The module system they likely already use in NixOS?

AndersonTorres · 2024-06-02T20:33:17Z

Yes, given that Nixpkgs is not tied to NixOS.

Atemu · 2024-06-03T17:14:52Z

What part do you expect to be hard to teach to newcomers? From the perspective of a user, they provide an attrset of configuration either way. I don't see how usage of this API would be any harder to understand than the status quo; using a module system requires basically no knowledge of its internal mechanisms.

I'd argue it'd be slightly simpler to use actually because error messages would be a lot easier to understand: "anonymous function at ... called with unexpected argument withFoo" vs. "the option foo does not exist". And that's ignoring the errors you could catch via typecheck instead of incredibly confusing "cannot coerce ... into a string".

AndersonTorres · 2024-06-07T01:34:54Z

What part do you expect to be hard to teach to newcomers?

Any part not well documented, to begin with.

https://nixos.org/manual/nixpkgs/stable/#module-system

Note

This chapter is new and not complete yet. For a gentle introduction to the module system, in the context of NixOS, see Writing NixOS Modules in the NixOS manual.

(bold mine)

Atemu · 2024-06-07T17:31:03Z

I do not see how the developer documentation for setting up a module system would be relevant to users merely using a module system that someone else already set up.

As per my previous comment, no newcomer needs to know about the module system's internal mechanisms in order to configure something with it.

nixos-discourse · 2024-06-09T06:28:44Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-there-a-way-to-get-the-description-of-a-package-rather-than-the-software-it-contains/46742/4

nixos-discourse · 2024-07-15T06:38:51Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/need-for-coorinated-enforced-boolean-switch-arguments-in-nixpkgs/48948/5

nixos-discourse · 2024-07-21T13:23:16Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/a-sad-state-of-nixpkgs/49397/5

Atemu · 2024-08-02T01:12:29Z

I just noticed that there has been prominent precedent for this in Nixpkgs for many years now: The structured Linux kernel config drv.

It uses much the same pattern as what I "came up" with. (I didn't actively think of the kernel config but it may have subconsciously influenced me.)

Introduced all the way back in 2018 by @teto: #42838

Now produces the same buildInputs (modulo order) as the regular ffmpeg

Atemu · 2024-09-12T02:16:14Z

Before:

$ hyperfine --warmup 10 -r 30 'nix-instantiate -A ffmpeg-headless'
Benchmark 1: nix-instantiate -A ffmpeg-headless
  Time (mean ± σ):     464.4 ms ±  15.9 ms    [User: 289.1 ms, System: 100.4 ms]
  Range (min … max):   429.5 ms … 486.9 ms    30 runs

$ hyperfine --warmup 10 -r 30 'nix-instantiate -A ffmpeg'
Benchmark 1: nix-instantiate -A ffmpeg
  Time (mean ± σ):     709.7 ms ±  33.3 ms    [User: 450.4 ms, System: 134.3 ms]
  Range (min … max):   649.3 ms … 750.7 ms    30 runs

$ hyperfine --warmup 10 -r 30 'nix-instantiate -A ffmpeg-full'
Benchmark 1: nix-instantiate -A ffmpeg-full
  Time (mean ± σ):     778.4 ms ±  36.8 ms    [User: 472.2 ms, System: 166.5 ms]
  Range (min … max):   710.5 ms … 828.2 ms    30 runs

After:

$ hyperfine --warmup 10 -r 30 'nix-instantiate -A ffmpeg-headless'
Benchmark 1: nix-instantiate -A ffmpeg-headless
  Time (mean ± σ):     484.4 ms ±  19.0 ms    [User: 316.3 ms, System: 92.1 ms]
  Range (min … max):   441.3 ms … 510.0 ms    30 runs

$ hyperfine --warmup 10 -r 30 'nix-instantiate -A ffmpeg'
Benchmark 1: nix-instantiate -A ffmpeg
  Time (mean ± σ):     727.0 ms ±  30.0 ms    [User: 459.6 ms, System: 148.2 ms]
  Range (min … max):   666.8 ms … 774.0 ms    30 runs

$ hyperfine --warmup 10 -r 30 'nix-instantiate -A ffmpeg-full'
Benchmark 1: nix-instantiate -A ffmpeg-full
  Time (mean ± σ):     815.5 ms ±  38.0 ms    [User: 523.3 ms, System: 157.6 ms]
  Range (min … max):   762.0 ms … 884.0 ms    30 runs

There appears to be an effect of about 5% but that only corresponds to ~1σ which isn't really significant, so it's hard to say whether there's an actual negative effect through runtime testing.

Here's the diff of NIX_SHOW_STATS=1 of ffmpeg-full which I expect to show the greatest difference since it contains 3 separate evals of ffmpeg:

│  1 │warning: you did not specify '--add-ro↵ │  1 │warning: you did not specify '--add-ro↵
│    │ot'; the result might be removed by th↵ │    │ot'; the result might be removed by th↵
│    │e garbage collector                     │    │e garbage collector
│  2 │/nix/store/3ykq8wg61ibq7xaxcz8h7ygf08i↵ │  2 │/nix/store/4pwzjp1bb4axh7r8ap3jdp777x0
│    │2l8zi-ffmpeg-full-6.1.1.drv!bin         │    │8a4k1-ffmpeg-full-6.1.1.drv!bin
│  3 │{                                       │  3 │{
│  4 │  "cpuTime": 0.46145400404930115,       │  4 │  "cpuTime": 0.5295500159263611,
│  5 │  "envs": {                             │  5 │  "envs": {
│  6 │    "bytes": 15570136,                  │  6 │    "bytes": 27651016,
│  7 │    "elements": 1173009,                │  7 │    "elements": 2054566,
│  8 │    "number": 773258                    │  8 │    "number": 1401811
│  9 │  },                                    │  9 │  },
│ 10 │  "gc": {                               │ 10 │  "gc": {
│ 11 │    "heapSize": 402915328,              │ 11 │    "heapSize": 402915328,
│ 12 │    "totalBytes": 195629904             │ 12 │    "totalBytes": 253786144
│ 13 │  },                                    │ 13 │  },
│ 14 │  "list": {                             │ 14 │  "list": {
│ 15 │    "bytes": 1935720,                   │ 15 │    "bytes": 3137688,
│ 16 │    "concats": 53881,                   │ 16 │    "concats": 78153,
│ 17 │    "elements": 241965                  │ 17 │    "elements": 392211
│ 18 │  },                                    │ 18 │  },
│ 19 │  "nrAvoided": 931140,                  │ 19 │  "nrAvoided": 1577220,
│ 20 │  "nrFunctionCalls": 693286,            │ 20 │  "nrFunctionCalls": 1228708,
│ 21 │  "nrLookups": 349445,                  │ 21 │  "nrLookups": 755979,
│ 22 │  "nrOpUpdateValuesCopied": 4800101,    │ 22 │  "nrOpUpdateValuesCopied": 5003831,
│ 23 │  "nrOpUpdates": 101294,                │ 23 │  "nrOpUpdates": 137413,
│ 24 │  "nrPrimOpCalls": 392853,              │ 24 │  "nrPrimOpCalls": 615520,
│ 25 │  "nrThunks": 1323307,                  │ 25 │  "nrThunks": 2033500,
│ 26 │  "sets": {                             │ 26 │  "sets": {
│ 27 │    "bytes": 96291744,                  │ 27 │    "bytes": 109203312,
│ 28 │    "elements": 5856951,                │ 28 │    "elements": 6502876,
│ 29 │    "number": 161283                    │ 29 │    "number": 322331
│ 30 │  },                                    │ 30 │  },
│ 31 │  "sizes": {                            │ 31 │  "sizes": {
│ 32 │    "Attr": 16,                         │ 32 │    "Attr": 16,

───────────────────────────────────────────────────────────────────────────────────────┐
• 35: warning: you did not specify '--add-root'; the result might be removed by the ga │
───────────────────────────────────────────────────────────────────────────────────────┘
│ 35 │    "Value": 24                         │ 35 │    "Value": 24
│ 36 │  },                                    │ 36 │  },
│ 37 │  "symbols": {                          │ 37 │  "symbols": {
│ 38 │    "bytes": 432883,                    │ 38 │    "bytes": 432598,
│ 39 │    "number": 41660                     │ 39 │    "number": 41663
│ 40 │  },                                    │ 40 │  },
│ 41 │  "values": {                           │ 41 │  "values": {
│ 42 │    "bytes": 47772936,                  │ 42 │    "bytes": 70875288,
│ 43 │    "number": 1990539                   │ 43 │    "number": 2953137
│ 44 │  }                                     │ 44 │  }
│ 45 │}                                       │ 45 │}

The only thing I take from this is that there are twice as many function calls now.

Slightly less nice but necessary because we pass packages individually

doronbehar · 2024-09-12T11:08:38Z

I don't think we have that many packages that would want to use this module system, hence I don't consider this price as too high for gradual insertion of this feature into Nixpkgs. I'd like to see this getting moved forward by rebasing etc, and see ofborg's checks will be green. I'd also like to practice afterwards in converting Gnuradio's and mpd's different features to use this module system.

I'd also like to discuss how the docs would look like.

Atemu · 2024-09-14T10:34:09Z

I'm confident ofBorgs checks will pass because this evals to the exact same result (modulo order).

You can simply pass it to make, no need for a custom checkPhase

Atemu · 2024-09-14T12:41:48Z

I've done a few more cleanups and refactors and I think I'm at a point where I'm quite happy with how the ffmpeg derivation turned out and can't immediately think of anything I'd change about the design anymore.

Atemu · 2024-09-14T12:44:46Z

pkgs/development/libraries/ffmpeg/generic.nix

    "--target_os=${if stdenv.hostPlatform.isMinGW then "mingw64" else stdenv.hostPlatform.parsed.kernel.name}"
    "--arch=${stdenv.hostPlatform.parsed.cpu.name}"
    "--pkg-config=${buildPackages.pkg-config.targetPrefix}pkg-config"


I've decided to keep these flags be managed manually; without implying them from the modules eval. Rationale being that it's just a few and they're not really something you'd usually (if ever) want to touch.

Forgot to actually delete it a few commits ago

Atemu · 2024-09-14T12:55:30Z

I'm pretty much on the same page w.r.t. eval time increase. For now, this only really makes sense to do in complex packages such as ffmpeg or Linux. A suspected eval time increase by ~1 standard deviation (10^-2 s) isn't a great cost to pay considering a typical NixOS eval takes 2-3 orders of magnitude longer.

While I'd like better documentation, the status quo is that nothing about package customisation is documented outside of code and its comments which is retained. I'd rather wait until we have a proper place and system coordinated for that at which point it should be trivial to plug ffmpeg's options into that.

What I'd like to discuss more is how to handle the override situation as it's all but intuitive or easy and I can't think of an easy solution. The best I can currently come up with is a dedicated override function (i.e. overrideFeatures).

pkgs/development/libraries/ffmpeg/generic.nix

doronbehar · 2024-09-15T15:06:35Z

What I'd like to discuss more is how to handle the override situation as it's all but intuitive or easy and I can't think of an easy solution. The best I can currently come up with is a dedicated override function (i.e. overrideFeatures).

Which would just be a thin wrapper around .override { configuration = ...; }?

While I'd like better documentation, the status quo is that nothing about package customisation is documented outside of code and its comments which is retained. I'd rather wait until we have a proper place and system coordinated for that at which point it should be trivial to plug ffmpeg's options into that.

It doesn't feel like a big deal to me - we have a place already to document how to use Nix expressions of packages, it's just not automatically generated. I agree we don't have to settle now on how exactly to build the workflow of adding these docs automatically, but I'd like to see that it is at least possible to generate some documentation files automatically from these expressions.

Atemu · 2024-09-15T16:40:30Z

Which would just be a thin wrapper around .override { configuration = ...; }?

It'd work as described in the OP.

Though implementing it could be tricky because an override functions' scope is not local to the package.

we have a place already to document how to use Nix expressions of packages

Where? That's news to me.

I'd like to see that it is at least possible to generate some documentation files automatically from these expressions.

This already works for NixOS, I don't see why it wouldn't work here aswell.

In fact, you can already introspect the documentation using a repl/nix-instantiate:

$ nix-instantiate --eval -A ffmpeg.eval.options.features.ass.description
"Control ass support in ffmpeg: (Advanced) SubStation Alpha subtitle rendering."

Atemu · 2024-09-15T16:55:11Z

This already works for NixOS, I don't see why it wouldn't work here aswell.

I tried it and it's trivial: pkgs.nixosOptionsDoc { inherit (ffmpeg.eval) options; }

What I noticed is that it puts a literal ‹name› where the submodule's name should be in sentences like The dependencies required to enable ‹name› support.. Not sure what's going on there.

doronbehar · 2024-09-16T08:58:46Z

Which would just be a thin wrapper around .override { configuration = ...; }?

It'd work as described in the OP.

Though implementing it could be tricky because an override functions' scope is not local to the package.

Hmm OK I see. Now I read in the OP that it is only an issue with multiple overrides? Why would someone need to do that?

Atemu · 2024-09-16T11:07:22Z

It's only an issue with composing overrides indeed but you never know what people might need to do with your derivation.

aanderse · 2024-09-16T11:11:43Z

multiple overrides are definitely a thing in nixos

consider a module which provides a package option watch requires a specific option - and allows the user to further customize options

i don't recall the concrete example to link at the moment but i if you look you will find it

Atemu · 2024-09-16T11:20:33Z

Steam does this for instance:

nixpkgs/nixos/modules/programs/steam.nix

Lines 45 to 65 in 8c9630d

    
           apply = steam: steam.override (prev: { 
        
             extraEnv = (lib.optionalAttrs (cfg.extraCompatPackages != [ ]) { 
        
               STEAM_EXTRA_COMPAT_TOOLS_PATHS = extraCompatPaths; 
        
             }) // (lib.optionalAttrs cfg.extest.enable { 
        
               LD_PRELOAD = "${pkgs.pkgsi686Linux.extest}/lib/libextest.so"; 
        
             }) // (prev.extraEnv or {}); 
        
             extraLibraries = pkgs: let 
        
               prevLibs = if prev ? extraLibraries then prev.extraLibraries pkgs else [ ]; 
        
               additionalLibs = with config.hardware.graphics; 
        
                 if pkgs.stdenv.hostPlatform.is64bit 
        
                 then [ package ] ++ extraPackages 
        
                 else [ package32 ] ++ extraPackages32; 
        
             in prevLibs ++ additionalLibs; 
        
             extraPkgs = p: (cfg.extraPackages ++ lib.optionals (prev ? extraPkgs) (prev.extraPkgs p)); 
        
           } // lib.optionalAttrs (cfg.gamescopeSession.enable && gamescopeCfg.capSysNice) 
        
           { 
        
             buildFHSEnv = pkgs.buildFHSEnv.override { 
        
               # use the setuid wrapped bubblewrap 
        
               bubblewrap = "${config.security.wrapperDir}/.."; 
        
             }; 
        
           });

As you can see, it also needs to take great care of overriding in a composable manner but that's only necessary when composing values of arguments; composing top-level arguments themselves without "deeper" values is trivial.

Darwin eval works now

Atemu added 4 commits May 17, 2024 12:28

ffmpeg: small first attempt

02e810b

mlterm: initial mini-module conversion

0d3005f

mlterm: further conversion

77762e4

mlterm: finish conversion using helper function

33ac366

Atemu requested review from roberth and doronbehar May 17, 2024 12:01

mlterm: fix all-packages override

8943dec

Atemu mentioned this pull request May 17, 2024

[RFC 0169] Feature parameter names NixOS/rfcs#169

Draft

doronbehar reviewed May 18, 2024

View reviewed changes

Atemu mentioned this pull request Jul 8, 2024

Defining variables with default values in package arguments can lead to problems #309892

Open

Atemu mentioned this pull request Aug 2, 2024

ffmpeg_7-full: Enable xev{d,e} on Darwin systems #328428

Merged

13 tasks

Atemu added 8 commits August 18, 2024 01:36

wip new options system with type

5676088

port buildInputs

3afb8b1

Now produces the same buildInputs (modulo order) as the regular ffmpeg

port configureFlags

698eafe

cleanup

39606cc

move variant description to option description

fc770c5

convert non-feature options

20a4a42

add back descriptions as first-class data entry

767516f

small improvements

42ae1cd

Atemu mentioned this pull request Sep 7, 2024

ffmpeg: remove withOgg #340217

Merged

13 tasks

Atemu added 2 commits September 12, 2024 05:50

cleanup module eval application

cff4b07

move options declaration to generic.nix

4383197

Slightly less nice but necessary because we pass packages individually

Atemu added 6 commits September 14, 2024 13:18

get rid of with aliases

726b8b5

get rid of remaining with* variables

40259ce

small refactors

218ef87

refactor check env var

9ec6044

You can simply pass it to make, no need for a custom checkPhase

fix formatting

297dd92

remove debug passthru

ccb7b7f

Atemu commented Sep 14, 2024

View reviewed changes

actually delete options.nix

2ba0097

Forgot to actually delete it a few commits ago

tobim reviewed Sep 14, 2024

View reviewed changes

pkgs/development/libraries/ffmpeg/generic.nix Show resolved Hide resolved

merge ffmpegFlags with default values

a0a07ac

AndersonTorres mentioned this pull request Sep 16, 2024

bochs: configure it via module system #342188

Draft

13 tasks

Atemu added 4 commits September 16, 2024 14:49

make description internal

c2afc38

improve descriptions

ba6e0f9

missing alsa gate

1dd2070

Darwin eval works now

qt-faststart only in full

4560290

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

package overrides using a mini module system #312432

package overrides using a mini module system #312432

Atemu commented May 17, 2024 •

edited

Loading

roberth commented May 17, 2024

doronbehar left a comment

AndersonTorres commented Jun 2, 2024

Atemu commented Jun 2, 2024

AndersonTorres commented Jun 2, 2024

Atemu commented Jun 3, 2024

AndersonTorres commented Jun 7, 2024

Atemu commented Jun 7, 2024

nixos-discourse commented Jun 9, 2024

nixos-discourse commented Jul 15, 2024

nixos-discourse commented Jul 21, 2024

Atemu commented Aug 2, 2024

Atemu commented Sep 12, 2024

doronbehar commented Sep 12, 2024

Atemu commented Sep 14, 2024

Atemu commented Sep 14, 2024

Atemu Sep 14, 2024

Atemu commented Sep 14, 2024

doronbehar commented Sep 15, 2024

Atemu commented Sep 15, 2024

Atemu commented Sep 15, 2024

doronbehar commented Sep 16, 2024

Atemu commented Sep 16, 2024

aanderse commented Sep 16, 2024

Atemu commented Sep 16, 2024

package overrides using a mini module system #312432

Are you sure you want to change the base?

package overrides using a mini module system #312432

Conversation

Atemu commented May 17, 2024 • edited Loading

Description of changes

Things done

roberth commented May 17, 2024

doronbehar left a comment

Choose a reason for hiding this comment

AndersonTorres commented Jun 2, 2024

Atemu commented Jun 2, 2024

AndersonTorres commented Jun 2, 2024

Atemu commented Jun 3, 2024

AndersonTorres commented Jun 7, 2024

Atemu commented Jun 7, 2024

nixos-discourse commented Jun 9, 2024

nixos-discourse commented Jul 15, 2024

nixos-discourse commented Jul 21, 2024

Atemu commented Aug 2, 2024

Atemu commented Sep 12, 2024

doronbehar commented Sep 12, 2024

Atemu commented Sep 14, 2024

Atemu commented Sep 14, 2024

Atemu Sep 14, 2024

Choose a reason for hiding this comment

Atemu commented Sep 14, 2024

doronbehar commented Sep 15, 2024

Atemu commented Sep 15, 2024

Atemu commented Sep 15, 2024

doronbehar commented Sep 16, 2024

Atemu commented Sep 16, 2024

aanderse commented Sep 16, 2024

Atemu commented Sep 16, 2024

Atemu commented May 17, 2024 •

edited

Loading