Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: Improve overlays section #255132

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 132 additions & 35 deletions doc/using/overlays.chapter.md
Original file line number Diff line number Diff line change
@@ -1,69 +1,166 @@
# Overlays {#chap-overlays}

This chapter describes how to extend and change Nixpkgs using overlays. Overlays are used to add layers in the fixed-point used by Nixpkgs to compose the set of all packages.
This chapter describes how to extend and change Nixpkgs using overlays.

Nixpkgs can be configured with a list of overlays, which are applied in order. This means that the order of the overlays can be significant if multiple layers override the same package.
TODO: Maybe add link to `lib.extends`/`lib.fix`

## Installing overlays {#sec-overlays-install}
## What are overlays {#sec-overlays-definition}
Copy link
Contributor

@hsjobeki hsjobeki Aug 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the important question is "what can i do with overlays". The technical correct definition is not as important in the first sentence.

Suggested change
## What are overlays {#sec-overlays-definition}
## What are overlays {#sec-overlays-definition}
Overlays can be used in general to extend or change existing attribute sets. Common applications are:
- Adding/extending packages to a locally imported version of nixpkgs
- Overriding packages
- Overriding packageSets
- Extending fixedPoint packageSets such as nixpkgs. (We should maybe explain somewhere what those fixed point package sets are.)


The list of overlays can be set either explicitly in a Nix expression, or through `<nixpkgs-overlays>` or user configuration files.
Overlays are Nix functions which accept two arguments, conventionally called `self` and `super`, and return a set of attributes. For example, the following is a valid overlay.

### Set overlays in NixOS or Nix expressions {#sec-overlays-argument}
```nix
self: super:

On a NixOS system the value of the `nixpkgs.overlays` option, if present, is passed to the system Nixpkgs directly as an argument. Note that this does not affect the overlays for non-NixOS operations (e.g. `nix-env`), which are [looked up](#sec-overlays-lookup) independently.
{
boost = super.boost.override {
python = self.python3;
};
rr = super.callPackage ./pkgs/rr {
stdenv = self.stdenv_32bit;
};
}
```

The list of overlays can be passed explicitly when importing nixpkgs, for example `import <nixpkgs> { overlays = [ overlay1 overlay2 ]; }`.
The first argument (`self`) corresponds to the final package set. You should use this set for the dependencies of all packages specified in your overlay. For example, all the dependencies of `rr` in the example above come from `self`, as well as the overridden dependencies used in the `boost` override.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The first argument (`self`) corresponds to the final package set. You should use this set for the dependencies of all packages specified in your overlay. For example, all the dependencies of `rr` in the example above come from `self`, as well as the overridden dependencies used in the `boost` override.
The first argument (`final`) corresponds to the final package set after all overlays have been applied. You should use this set for the dependencies of all packages specified in your overlay. For example, all the dependencies of `rr` in the example above come from `final`, as well as the overridden dependencies used in the `boost` override.


NOTE: DO NOT USE THIS in nixpkgs. Further overlays can be added by calling the `pkgs.extend` or `pkgs.appendOverlays`, although it is often preferable to avoid these functions, because they recompute the Nixpkgs fixpoint, which is somewhat expensive to do.
The second argument (`super`) corresponds to the result of the evaluation of the previous stages of Nixpkgs. It does not contain any of the packages added by the current overlay, nor any of the following overlays. This set should be used either to refer to packages you wish to override, or to access functions defined in Nixpkgs. For example, the original recipe of `boost` in the above example, comes from `super`, as well as the `callPackage` function.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The second argument (`super`) corresponds to the result of the evaluation of the previous stages of Nixpkgs. It does not contain any of the packages added by the current overlay, nor any of the following overlays. This set should be used either to refer to packages you wish to override, or to access functions defined in Nixpkgs. For example, the original recipe of `boost` in the above example, comes from `super`, as well as the `callPackage` function.
The second argument (`prev`) corresponds to the result of the evaluation of all previous overlays. It does not contain any of the packages added by the current overlay, nor any of the following overlays. This set should be used either to refer to packages you wish to override, to access functions defined in Nixpkgs, or simply if the addition made needs the original unmodified value. For example, the original recipe of `boost` in the above example, comes from `prev`, as well as the `callPackage` function.


### Install overlays via configuration lookup {#sec-overlays-lookup}
The value returned by this function should be a set similar to `pkgs/top-level/all-packages.nix`, containing overridden and/or new packages.

The list of overlays is determined as follows.
Overlays are similar to other methods for customizing Nixpkgs, in particular the `packageOverrides` attribute described in [](#sec-modify-via-packageOverrides). Indeed, `packageOverrides` acts as an overlay with only the `super` argument. It is therefore appropriate for basic use, but overlays are more powerful and easier to distribute.

1. First, if an [`overlays` argument](#sec-overlays-argument) to the Nixpkgs function itself is given, then that is used and no path lookup will be performed.
Note that the order of the overlays can matter if multiple overlays set the same attributes.

2. Otherwise, if the Nix path entry `<nixpkgs-overlays>` exists, we look for overlays at that path, as described below.
## Applying overlays {#sec-overlays-applying}

See the section on `NIX_PATH` in the Nix manual for more details on how to set a value for `<nixpkgs-overlays>.`
In order to define overlays, you need to know where to put them so they get applied.
See [here](#sec-overlays-definition) for how to define overlays.

3. If one of `~/.config/nixpkgs/overlays.nix` and `~/.config/nixpkgs/overlays/` exists, then we look for overlays at that path, as described below. It is an error if both exist.
There are various ways of applying overlays, but by default none are applied.

If we are looking for overlays at a path, then there are two cases:
### Using nix expression {#d}

- If the path is a file, then the file is imported as a Nix expression and used as the list of overlays.
#### Applying overlays when importing Nixpkgs {#sec-overlays-applying-nixpkgs}

- If the path is a directory, then we take the content of the directory, order it lexicographically, and attempt to interpret each as an overlay by:
When importing Nixpkgs, a list of overlays can be specified using the `overlays` attribute in the argument:
```nix
pkgs = import <nixpkgs> {
overlays = [
firstOverlay
secondOverlay
];
}
```

- Importing the file, if it is a `.nix` file.
If the `overlays` argument is not passed, these defaults are used:

- Importing a top-level `default.nix` file, if it is a directory.
- If the [path expression](https://nixos.org/manual/nix/stable/language/values#type-path) `<nixpkgs-overlays>` points to an existing path, we look for overlays there, as described in [the below section](#sec-overlays-applying-nixpkgs-paths).

Because overlays that are set in NixOS configuration do not affect non-NixOS operations such as `nix-env`, the `overlays.nix` option provides a convenient way to use the same overlays for a NixOS system configuration and user configuration: the same file can be used as `overlays.nix` and imported as the value of `nixpkgs.overlays`.
Where such path expressions point is defined with the [`NIX_PATH`](https://nixos.org/manual/nix/stable/command-ref/env-common.html#env-NIX_PATH) environment variable.

On a NixOS system, this variable can be set using the [`nix.nixPath` option](https://search.nixos.org/options?show=nix.nixPath&query=nix.nixPath).
On non-NixOS systems, see [here](https://wiki.archlinux.org/title/Environment_variables) for how to set environment variables.

- If either `~/.config/nixpkgs/overlays.nix` or `~/.config/nixpkgs/overlays/` exist, then we look for overlays at that path, as described in [the below section](#sec-overlays-applying-nixpkgs-paths).
It is an error if both exist.

##### How paths get turned into overlays {#sec-overlays-applying-nixpkgs-paths}

- If the path is a file, then the file is imported as a Nix expression and used as the list of overlays.
Here is an example:
```nix
# overlays.nix
[
# first overlay
(final: prev: { ... })

# second overlay
(final: prev: { ... })
]
```

- If the path is a directory, then we take the entries of the directory, order them alphabetically, and attempt to interpret each as an overlay by:

## Defining overlays {#sec-overlays-definition}
- Importing the file, if it is a `.nix` file.

Overlays are Nix functions which accept two arguments, conventionally called `self` and `super`, and return a set of packages. For example, the following is a valid overlay.
- Importing a top-level `default.nix` file, if it is a directory.

Here's an example of a suitable directory structure:
```
overlays
├── 50-firstOverlay.nix
└── 60-secondOverlay
└── default.nix
```

#### Applying further overlays on top of an imported Nixpkgs {#c}

If you already have an imported Nixpkgs (often in a variable called `pkgs`),
you can apply a list of overlays on top using `pkgs.appendOverlays`:

```nix
self: super:
pkgsModified = pkgs.appendOverlays [
firstAdditionalOverlay
secondAdditionalOverlay
]
```

{
boost = super.boost.override {
python = self.python3;
};
rr = super.callPackage ./pkgs/rr {
stdenv = self.stdenv_32bit;
};
}
Or just a single overlay using `pkgs.extend`:
```nix
pkgsModified = pkgs.extend additionalOverlay
```

The first argument (`self`) corresponds to the final package set. You should use this set for the dependencies of all packages specified in your overlay. For example, all the dependencies of `rr` in the example above come from `self`, as well as the overridden dependencies used in the `boost` override.
Both the overlays of original `pkgs` and the new ones will be applied, in this order.

The second argument (`super`) corresponds to the result of the evaluation of the previous stages of Nixpkgs. It does not contain any of the packages added by the current overlay, nor any of the following overlays. This set should be used either to refer to packages you wish to override, or to access functions defined in Nixpkgs. For example, the original recipe of `boost` in the above example, comes from `super`, as well as the `callPackage` function.
:::{.warning}
The `pkgsModified` result cannot re-use any values already computed for the `pkgs` values,
so these functions may be expensive.

The value returned by this function should be a set similar to `pkgs/top-level/all-packages.nix`, containing overridden and/or new packages.
This should not be used in Nixpkgs.
:::

### Commands implicitly importing Nixpkgs {#b}

These CLI commands can implicitly import Nixpkgs without it being specified in a Nix expression.

#### `nixos-rebuild` {#e}

NixOS implicitly imports Nixpkgs for you [with the `overlays` argument](#sec-overlays-applying-nixpkgs) set to the value of the [`nixpkgs.overlays`](https://search.nixos.org/options?show=nixpkgs.overlays&query=nixpkgs.overlays) option.

This option is empty by default.

This is the only thing that the `nixpkgs.overlays` option affects.
It does not propagate the overlays to the non-NixOS commands.

Note that this does not affect the overlays for non-NixOS operations (e.g. `nix-env`), which are looked up independently, see the following sections.

#### `nix-shell -p` {#f}

`nix-shell -p` implicitly imports Nixpkgs without any arguments, such that the defaults from <link section> apply.

#### `nix-build '<nixpkgs>'` / `nix-env -f '<nixpkgs>'` {#g}

`nix-build` and `nix-env -f` with any Nixpkgs path, such as `<nixpkgs>` or `~/src/nixpkgs`, ...

Will by default pass no arguments to the Nixpkgs imports, therefore the defaults from <link section> apply.

The `overlays` argument can be overridden using the [`--arg`](<link to Nix manual>) option, for example to not use the default overlays:
```
nix-build '<nixpkgs>' --arg overlays '[ ]'
nix-env -f '<nixpkgs>' --arg overlays '[ ]'
```

## Sharing overlays between a NixOS configuration and nix commands {#a}

TODO: Rewrite
Because overlays that are set in NixOS configuration do not affect non-NixOS operations such as `nix-env`, the `overlays.nix` option provides a convenient way to use the same overlays for a NixOS system configuration and user configuration: the same file can be used as `overlays.nix` and imported as the value of `nixpkgs.overlays`.

```nix
# configuration.nix

nixpkgs.overlays = import ~/.config/nixpkgs/overlays.nix;
```

Overlays are similar to other methods for customizing Nixpkgs, in particular the `packageOverrides` attribute described in [](#sec-modify-via-packageOverrides). Indeed, `packageOverrides` acts as an overlay with only the `super` argument. It is therefore appropriate for basic use, but overlays are more powerful and easier to distribute.

## Using overlays to configure alternatives {#sec-overlays-alternatives}

Expand Down
Loading