python3Packages.mkPythonEditablePackage: init

[adisbladis]

Description of changes

When developing Python packages it's common to install packages in editable mode.
Like mkPythonMetaPackage this function exists to create an otherwise empty package, but also containing a pointer to an impure location outside the Nix store that can be changed without rebuilding.

The editable root is passed as a string. Normally .pth files contains absolute paths to the mutable location. This isn't always ergonomic with Nix, so I have decided to expand environment variables at runtime.
This means that a shell hook setting up something like a $REPO_ROOT variable can be used as the relative package root.

Note that overriding packages deeper in the dependency graph can work, but it's not the primary usecase and overriding existing packages can make others break in unexpected ways.

{ pkgs ? import <nixpkgs> { } }:

let
  myPython = pkgs.python.override {
    self = myPython;
    packageOverrides = pyfinal: pyprev: {
      # An editable package with a script that loads our mutable location
      my-editable = pyfinal.mkPythonEditablePackage {
        pname = "my-editable";
        version = "0.1.0";
        root = "$REPO_ROOT/src"; # Use environment variable expansion at runtime
        # Inject a script (other PEP-621 entrypoints are also accepted)
        scripts = {
          my-script = "my_editable.main:main";
        };
      };
    };
  };

  pythonEnv =  testPython.withPackages (ps: [ ps.my-editable ]);

in pkgs.mkShell {
  packages = [ pythonEnv ];
}

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[phaer]

Thank you for this!

Think i found a typo, commented a bit on docs and found a single semantic change I'd like to see: prepending, not appending to sys.path.

[adisbladis]

Unless someone gives me a good reason not to I intend to self-merge this sooner rather than later.

[natsukium]

I'm not sure if adding more buildPythonPackage friends is the direction we should go.
Can we take an approach that injects into existing builders, maybe editableInstallHook?

[adisbladis]

I'm not sure if adding more buildPythonPackage friends is the direction we should go.
Can we take an approach that injects into existing builders, maybe editableInstallHook?

No. We need to create package metadata and pointer files in the store as regular Python packages, and these should be built without sources.
The metadata files needs to be consumed by creating a venv-like created by withPackages.

There is no place to set up a hook to get editable behaviour for development shells.

[adisbladis]

I could conjure up an implementation of a hook-based editable builder, but it's usage would have to look like:

buildPythonPackage {
  pname = "my-editable";
  version = "0.1.0";

  # Assuming that you'd want to use pyproject.toml from current directory, don't know if there is a more elegant way to express this
  # If you'd forget to filter your sources the point of the editable is kinda moot since you'd rebuild all the time anyway.
  src = lib.filesets.toSource {
    root = ./.;
    fileset = [ ./pyproject.toml ];
  };

  pyproject = true;

  editableRoot = "$REPO_ROOT";

  nativeBuildInputs = [
    editablePatchHook # Patches pyproject.toml with the editable root build and forces hatchling to be the build-system
  ];
}

That's a lot of stuff to get right every time for every user. I prefer a function abstraction based approach in this instance, even though I generally agree that we should move towards hooks.

[adisbladis]

Giving a 24 hour notice to merge.

[natsukium]

Is the primary concern the handling of src?
A hook-based interface looks good, but I agree that it could be confusing to the end user.
I'm leaning towards having a simple function since it's for external use.

[adisbladis]

Is the primary concern the handling of src?

That's one of the main two concerns, but also handling of editableRoot. It's very appealing to go for editableRoot = ./. or something similar, which will end up pointing to a store path, and not your editable location.

By making it a hook it's not possible to enforce the type of editableRoot at eval time.
We could of course capture this at build time and error out there instead, but I find the user experience of that terrible compared to erroring out at eval time.

I'm leaning towards having a simple function since it's for external use.

I think this is really the core of it. This is a development helper, not a composable builder.

[natsukium]

Thanks for your explanation. I support this implementation.

[phaer]

I think this a good step forward, thank you!