Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: how to build a package without "cruft"? #80

Open
jfly opened this issue Dec 4, 2024 · 2 comments
Open

Question: how to build a package without "cruft"? #80

jfly opened this issue Dec 4, 2024 · 2 comments

Comments

@jfly
Copy link
Contributor

jfly commented Dec 4, 2024

The hello-world example provides a package like this:

packages.x86_64-linux.default = pythonSet.mkVirtualEnv "hello-world-env" workspace.deps.default;

If I nix build this, I get the following result/:

$ tree result/
result/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── Activate.ps1
│   ├── python -> python3.12
│   ├── python3 -> python3.12
│   └── python3.12 -> /nix/store/zv1kaq7f1q20x62kbjv6pfjygw5jmwl6-python3-3.12.7/bin/python3.12
├── include
│   └── python3.12
├── lib
│   └── python3.12
│       └── site-packages
│           ├── hello.py -> /nix/store/pvn93v171zf9z45pxvyyjz553kqym3nw-tmp-njenrenxze-0.1.0/lib/python3.12/site-packages/hello.py
│           ├── __pycache__ -> /nix/store/pvn93v171zf9z45pxvyyjz553kqym3nw-tmp-njenrenxze-0.1.0/lib/python3.12/site-packages/__pycache__
│           └── tmp_njenrenxze-0.1.0.dist-info -> /nix/store/pvn93v171zf9z45pxvyyjz553kqym3nw-tmp-njenrenxze-0.1.0/lib/python3.12/site-packages/tmp_njenrenxze-0.1.0.dist-info
├── lib64 -> lib
├── nix-support
└── pyvenv.cfg

All the venv stuff in my result/bin directory feels unnecessary (in a less trivial project, it will even include scripts from my project's transitive dependencies). The include/, lib*/, nix-support, and pyvenv.cfg stuff also feels unnecessary to me.

Based on my reading of pyproject-nix, I suspect the way to do this is to invoke lib.renderers.buildPythonPackage + nixpkgs's buildPythonApplication. I tried massaging the hello-world example into doing just this, and I ended up with this:

packages.default = pythonSet.python.pkgs.buildPythonApplication (
  inputs.pyproject-nix.lib.renderers.buildPythonPackage {
    project = inputs.pyproject-nix.lib.project.loadUVPyproject {
      projectRoot = ../.;
    };
    python = pythonSet.python;
    pythonPackages = pythonSet;
  }
);

This seems to work, but feels kind of tedious and a little brittle (I've now got my projectRoot specified in 2 places). Is this the right approach? Is there a better way? (Happy to send in a PR if you point me in the right direction)

@adisbladis
Copy link
Member

I think the way to structure application builds is not to eliminate the "cruft" but to hide it in another wrapper layer.

Conceptually like:

let
  venv = pythonSet.mkVirtualEnv "hello-world-env" workspace.deps.default;
in 
  pkgs.runCommand "hello-world" { } ''
    mkdir -p $out/bin
    ln -s ${venv}/bin/hello $out/bin
  ''

There should be a utility function to do that:

let
  venv = pythonSet.mkVirtualEnv "hello-world-env" workspace.deps.default;
in
  mkApplication {
    inherit venv;
    from = pythonSet.hello;
  }

This hypothetical function would take a built environment, and only extract relevant bits given a package & a venv.
Not sure about the interface. The function above is only for illustrative purposes.

All the venv stuff in my result/bin directory feels unnecessary (in a less trivial project, it will even include scripts from my project's transitive dependencies). The include/, lib*/, nix-support, and pyvenv.cfg stuff also feels unnecessary to me.

At some point you need an environment that lives somewhere.
Building individual applications from a Python dependency graph perspective makes little semantic sense. They're not runnable without being a part of an environment.
Nixpkgs hacks around this by abusing $PYTHONPATH & it's wrapper equivalent $NIX_PYTHONPATH, but this is not a good solution.

Based on my reading of pyproject-nix, I suspect the way to do this is to invoke lib.renderers.buildPythonPackage + nixpkgs's buildPythonApplication. I tried massaging the hello-world example into doing just this, and I ended up with this:

pyproject.nix supports nixpkgs python builders, but uv2nix isn't using them at all.
Equivalents of lib.renderers for the pyproject.nix build infra are documented in https://pyproject-nix.github.io/pyproject.nix/build/lib/renderers.html, but you don't have to bother with this when using uv2nix.

@jfly
Copy link
Contributor Author

jfly commented Dec 5, 2024

I think the way to structure application builds is not to eliminate the "cruft" but to hide it in another wrapper layer.

Yes, totally. For me, I think I'd want a helper that symlinks just the scripts of a given pyproject (the "hypothetical function ... [to] ... extract relevant bits given a package & a venv" you mentioned above).

Feel free to close this, or leave this open to track implementing that hypothetical function.

At some point you need an environment that lives somewhere.

Yes, totally. I meant that I want a wrapper on top of the venv. I don't mind the fact that there's a venv derivation realized in my /nix/store ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants