Recipe for bringing local wheels into Pixi

[liquidcarbon]

There's a non-public package that is being developed internally on a shared server that uses poetry. It does not exist anywhere on the internet. It is updated at least once a week.

I am starting another package that will use the first package. There is a significant overlap in dependencies between the two: the first is a data processor and a FastAPI app, the second will be an orchestration package (Metaflow or Prefect) that will use the above processor.

How would you suggest to construct Pixi environment that makes it easy to stay current of the first package?
Can uv or rip help?

The only thing I'm thinking of now is to define and run as needed a Pixi task that would call something like pip install --no-index --find-links /path/to/wheels/ package_name, which seems likely to introduce dependency conflicts.

There's probably more than one sensible option. Hoping to keep this discussion evergreen. To be honest, I don't know how to properly bring in local wheels into Poetry or Conda or any other tool, so please suggest non-Pixi options too, so that we can conclude in the end that Pixi is the best. :)

[ruben-arts]

He @liquidcarbon,

TL;DR: It's coming soon, tested it on Tuesday and the basis works. PR will be open soon! De-serialization of the pixi.toml is already in:

pixi/src/project/manifest/python.rs

Lines 137 to 154 in c26c897

	pub enum PyPiRequirementType {
	Git {
	git: Url,
	branch: Option<String>,
	tag: Option<String>,
	rev: Option<String>,
	subdirectory: Option<String>,
	},
	Path {
	path: PathBuf,
	editable: Option<bool>,
	},
	Url {
	url: Url,
	},
	// Always try last to avoid serializing as version when it is not.
	Version(VersionOrStar),
	}

That said, I'm not sure how flexible this project is but I would advice to try to build a conda package from that package using our rattler-build, you could upload it to our prefix.dev/channels and use it from there. Or push it to a self hosted Quetz server, if you want to use it on-prem.
This will give the best and fastest pixi experience.

[liquidcarbon]

Oof can you pls explain how it helps use a preexisting wheel?
It's most likely not possible to publish this package.

[pavelzw]

Ideally you have a private conda registry hosted somewhere. Quetz is a self-hosted alternative. You can also create your own anaconda.org channel or prefix.dev channel if you have access to the internet.

[liquidcarbon]

Surely people have been co-developing interdependent packages before. There's got to be more than one right answer, and I wonder what they did, and how Pixi can help.

Private conda channel may be a great option that I can't fully appreciate, because I don't have experience there. May I be difficult for a moment and say "I don't want to?"

What other options exist?

[ruben-arts]

Currently (release v0.16.1) you'll have to resort to pip to do.whl installation. I understand this is not the most comfortable solution, native support for .whl files is build as we speak.

[liquidcarbon]

No problem, I bet the wishlist is long :)

[synapticarbors]

Maybe I'm not fully understanding your requirements, but could you just run:

pip install --no-index --no-deps --find-links /path/to/wheels/ package_name

either in a task (see https://github.com/prefix-dev/pixi/blob/main/examples/docker/pixi.toml#L27 or https://github.com/prefix-dev/pixi/blob/main/examples/docker/pixi.toml#L31) or in an activated pixi shell? The --no-deps flag should keep pip from pulling in dependencies and just use what you've defined in your environment.

I use some variant of this all of the time either in pixi or using mamba in a conda env. The only assumption is that the wheel files (or sdist) are being built my some other outside process. I guess finding whl files outside of the pixi project violates some ideas about reproducibility and isolation, but it's not an actual issue for development. I tend to build the whl and then copy it into the project directory that way if it's missing, it's clear that it should have been local, rather than depending on identical directory structures outside of the project.

[liquidcarbon]

Nice, I was also thinking along the lines of a post-install task, but was not aware of --no-deps!
What are the effects of these flags?

  --no-index                  Ignore package index (only looking at --find-links URLs instead).
  -f, --find-links <url>      If a URL or path to an html file, then parse for links to archives such as sdist
                              (.tar.gz) or wheel (.whl) files. If a local path or file:// URL that's a directory, then
                              look for archives in the directory listing. Links to VCS project URLs are not supported.

[liquidcarbon]

I'm imagining a future...

pixi.toml
---
[dependencies]
python = ">=3.12"
x
z
/path/to/requirements.txt
/path/to/package.whl

suppose the requirements file asks for packages a, x, y
and the wheel for a, b, z

Union these three set of requirements and you get

[dependencies]
python = ">=3.12"
a
b
x
y
z

Wouldn't it be convenient? and not too difficult to implement?

Conda supports something similar, though it's not well-documented. https://stackoverflow.com/questions/57835528/how-to-install-local-library-with-pip-to-a-conda-environment-using-environtment
https://github.com/conda/conda/blob/fd89e90b81b196f9aff6a3971b2730da85216aa7/tests/conda_env/support/advanced-pip/environment.yml

[pavelzw]

Dynamically loading in requirement files comes with a tail of potential problems. This would go against pixi's vision of isolation and reproducibility.

[liquidcarbon]

After pixi global install uv (#992) I'm experimenting with pixi tasks and uv :

[tasks]
uvadd = "uv pip install $1 --python $(which python)"

[dependencies]
python = "3.11.*"

The --python $(which python) is the pointer to python folders created by Pixi
that enables:

• pixi run uvadd 'pkg @ /path/pkg/dist/pkg.whl' --dry-run (installs pkg and its dependencies)
• pixi run uvadd 'pkg @ $(ls -1 /path/pkg/dist/*.whl | sort -V | tail -1)' --dry-run (installs latest wheels if several are sitting in the folder)
• pixi run uvadd 'pkg @ /path/pkg/' --dry-run (looks for setup.py or pyproject.toml in the given folder; installs the package's dependencies but not the package itself, sort of inverse of pip install --no-deps)
  (--dry-run just to see what happens at first)

---

starting from scratch (only pixi.toml like the above), this whole thing takes ~10-15 sec. Not bad!

pixi run uvadd 'mypkg @ /.../mypkg-1.1.28-py3-none-any.whl'
✨ Pixi task (default): uv pip install $1 --python $(which python) 'mypkg @ /.../mypkg-1.1.28-py3-none-any.whl' 
Resolved 65 packages in 260ms
Downloaded 37 packages in 4.47s
Installed 65 packages in 339ms
 + anyio==4.3.0
 + asttokens==2.4.1
 + boto3==1.34.64
 + botocore==1.34.64
 + certifi==2024.2.2
 + cffi==1.16.0
 + charset-normalizer==3.3.2
 + click==8.1.7
 + cloudpathlib==0.16.0
 + cramjam==2.8.2
 + cryptography==42.0.5
 + decorator==5.1.1
 + duckdb==0.9.2
 + et-xmlfile==1.1.0
 + executing==2.0.1
 + fastapi==0.95.2
 + fastparquet==2023.10.1
 + fsspec==2024.3.0
 + greenlet==3.0.3
 + h11==0.14.0
 + idna==3.6
 + ipython==8.22.2
 + itables==1.7.1
 + jedi==0.19.1
 + jmespath==1.0.1
 + loguru==0.7.2
 + matplotlib-inline==0.1.6
 + msal==1.27.0
 + numpy==1.26.4
 + office365-rest-python-client==2.5.6
 + openpyxl==3.1.2
 + packaging==24.0
 + pandas==2.2.1
 + parso==0.8.3
 + pexpect==4.9.0
 + prompt-toolkit==3.0.43
 + ptyprocess==0.7.0
 + pure-eval==0.2.2
 + pycparser==2.21
 + pydantic==1.10.14
 + pygments==2.17.2
 + pyjwt==2.8.0
 + pymssql==2.2.11
 + pyrsistent==0.20.0
 + python-dateutil==2.9.0.post0
 + python-dotenv==1.0.1
 + python-multipart==0.0.6
 + python-ulid==1.1.0
 + pytz==2024.1
 + requests==2.31.0
 + s3transfer==0.10.1
 + six==1.16.0
 + smart-open==6.4.0
 + sniffio==1.3.1
 + sqlalchemy==2.0.28
 + stack-data==0.6.3
 + starlette==0.27.0
 + mypkg==1.1.28 (from file:///...mypkg-1.1.28-py3-none-any.whl)
 + traitlets==5.14.2
 + typing-extensions==4.10.0
 + tzdata==2024.1
 + urllib3==2.2.1
 + uvicorn==0.22.0
 + wcwidth==0.2.13
 + xmltodict==0.13.0

Feels like I'm cheating again, like with poetry run pip install

[tdejager]

So with the source dependencies PR we are developing (which will need some refinement when it lands) you can at least point to a wheel file directly in the pypi-requirements section 😀 either through a url or a file path.

See: #985