Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch from hatch to uv for package and environment management #3577

Open
binste opened this issue Sep 6, 2024 · 8 comments · May be fixed by #3723
Open

Switch from hatch to uv for package and environment management #3577

binste opened this issue Sep 6, 2024 · 8 comments · May be fixed by #3723
Labels
enhancement help wanted

Comments

@binste
Copy link
Contributor

binste commented Sep 6, 2024 •
edited
Loading

What is your suggestion?

uv is now a Python packaging manager https://astral.sh/blog/uv-unified-python-packaging. For a while already it had a lot of useful functionality but now it has all the features we'd need to replace hatch. Paraphrasing @mattijn (correct me if I missed anything!), switching to uv might help us attract new developers. It's very likely that uv is much more popular than hatch, judging based on GitHub stars but also development activity and feature set. Due to this:

  • It makes it easier for new Altair contributors to get started as they might already know it from other projects
  • Or if a new contributor has not yet used it, it's likely that they are more interested in learning it

Have you considered any alternative solutions?

Poetry would be an alternative. uv seems closer to hatch so less of a switching cost and it just has a super fast solver and we have it already installed for the linting and formatting
@dangotbanned
Copy link
Member

dangotbanned commented Sep 6, 2024 •
edited
Loading

FYI we're already using uv for dependencies.

I'd support switching over fully if there was no loss of functionality.

A first step would be trying to replace pip and hatch in all the GitHub workflows.
They now have proper documentation - which was missing when I tried that out before

@joelostblom
Copy link
Contributor

I'd support switching over fully if there was no loss of functionality.

I'm also on board with this

@jonmmease
Copy link
Contributor

I think moving to uv sounds like a fine direction. One thing to throw out there is that I'm using pixi for VegaFusion, and will soon be using it for vl-convert. Pixi is a very similar idea, but has a few advantages for certain kinds of projects.

  • It can pull dependencies from PyPI using en embedded copy of uv, so this is just as fast as plain uv
  • It can also pull dependencies from conda-forge. This is really useful for non-python development dependencies. For example, in VegaFusion I configure Pixi to install Rust, Java, and Node.js from conda-forge so that these don't need to be installed at the system level.
  • Pixi includes a cross-platform shell (based on Deno's shell) that it uses to evaluate tasks. This shell includes a bunch of basic unix commands like (cp, rm, pwd, cat, etc.) and it has access to any CLI dependencies installed from conda-forge. So you can write tasks as if you're on linux, and they work the same on Windows.
  • Pixi tasks can depend on other tasks, and they support hash-based caching to intelligently skip running the parent tasks when not needed.

Altair development doesn't require any non-Python dependencies, so the advantage of Pixi over uv isn't that strong, but wanted to mention it as an alternative to at least consider.

@dangotbanned
Copy link
Member

@jonmmease I'm quite keen to use pixi in vl_convert as you know vega/vl-convert#186

Altair development doesn't require any non-Python dependencies, so the advantage of Pixi over uv isn't that strong, but wanted to mention it as an alternative to at least consider.

I would say the advantages you listed are interesting but not sure how they'd benefit altair.

Also from reading the docs, it seems to be missing building & publishing.
Not fully sure how that works in altair, but I thought hatch & uv both had these features?

@jonmmease
Copy link
Contributor

Yeah, for building a publishing you still use python -m build and twine. And I have no objection to using uv (or staying with hatch for that matter).

@dangotbanned
Copy link
Member

dangotbanned commented Sep 10, 2024 •
edited
Loading

FYI we're already using uv for dependencies.

I'd support switching over fully if there was no loss of functionality.

A first step would be trying to replace pip and hatch in all the GitHub workflows. They now have proper documentation - which was missing when I tried that out before

Related

@dangotbanned
Copy link
Member

Noting here that switching to uv from pip in build.yml may unblock https://github.com/vega/altair/actions/runs/10861487260/job/30143440337?pr=3591

If I've understood geopandas/pyogrio#450 correctly, installing pyogrio with conda instead of pip avoided the same error

@dangotbanned
Copy link
Member

I've recently been working with uv some more in https://github.com/vega/vega-datasets:

Definitely a joy to work with, but there is one gap we'd need to solve before the switch hatch -> uv

Either all of our hatch scripts need to be:

  1. Migrated into python scripts (for cross-platform support)
  2. Migrated to some other task runner
hatch scripts

altair/pyproject.toml

Lines 118 to 122 in 9002472

generate-schema-wrapper = [
"mypy tools",
"python tools/generate_schema_wrapper.py",
"test"
]

altair/pyproject.toml

Lines 131 to 143 in 9002472

update-init-file = [
"python tools/update_init_file.py",
"ruff check .",
"ruff format .",
]
test-fast = [
"ruff check .", "ruff format .",
"pytest -p no:randomly -n logical --numprocesses=logical --doctest-modules tests altair tools -m \"not slow\" {args}"
]
test-slow = [
"ruff check .", "ruff format .",
"pytest -p no:randomly -n logical --numprocesses=logical --doctest-modules tests altair tools -m \"slow\" {args}"
]

altair/pyproject.toml

Lines 145 to 162 in 9002472

[tool.hatch.envs.hatch-test]
# https://hatch.pypa.io/latest/tutorials/testing/overview/
features = ["all", "dev", "doc"]
# https://pytest-xdist.readthedocs.io/en/latest/distribution.html#running-tests-across-multiple-cpus
default-args = ["--numprocesses=logical","--doctest-modules", "tests", "altair", "tools"]
parallel = true
[[tool.hatch.envs.hatch-test.matrix]]
python = ["3.9", "3.10", "3.11", "3.12", "3.13"]
[tool.hatch.envs.hatch-test.scripts]
run = [
"ruff check .",
"ruff format --diff --check .",
"mypy altair tests",
"pytest{env:HATCH_TEST_ARGS:} {args}"
]
run-cov = "coverage run -m pytest{env:HATCH_TEST_ARGS:} {args}"
cov-combine = "coverage combine"
cov-report = "coverage report"

altair/pyproject.toml

Lines 167 to 196 in 9002472

[tool.hatch.envs.doc.scripts]
clean = "rm -rf doc/_build"
clean-generated = ["rm -rf doc/user_guide/generated", "rm -rf doc/gallery"]
clean-all = ["clean", "clean-generated", "rm -rf doc/_images"]
clean-win = "if exist doc\\_build rd /s /q doc\\_build"
clean-generated-win = [
"if exist doc\\user_guide\\generated rd /s /q doc\\user_guide\\generated",
"if exist doc\\gallery rd /s /q doc\\gallery",
]
clean-all-win = [
"clean-win",
"clean-generated-win",
"if exist doc\\_images rd /s /q doc\\_images",
]
build-html = [
"mkdir -p doc/_images",
"sphinx-build -b html -d doc/_build/doctrees doc doc/_build/html",
]
build-html-win = [
"if not exist doc\\_images md doc\\_images",
"sphinx-build -b html -d doc\\_build\\doctrees doc doc\\_build\\html",
]
doctest = "sphinx-build -b doctest -d doc/_build/doctrees doc doc/_build/doctest"
coverage = "sphinx-build -b coverage -d doc/_build/doctrees doc doc/_build/coverage"
serve = "(cd doc/_build/html && python -m http.server)"
publish-clean-build = [
"clean-all",
"build-html",
"(cd doc && bash sync_website.sh)",
]

It makes it easier for new Altair contributors to get started as they might already know it from other projects
@binste

I think having all these scripts/commands fragmented between files - rather than centralised in pyproject.toml - might make it a little more difficult to get started.

Important

This is my only concern. I really want to switch to uv as soon as we can solve this

@dangotbanned dangotbanned changed the title Switch from hatch to uv for package and environment management Switch from hatch to uv for package and environment management Dec 17, 2024
This was referenced Dec 20, 2024
Closed
Merged
dangotbanned added a commit that referenced this issue Dec 25, 2024
@dangotbanned
ci: replace sync_website.sh with .py
ddc8358 
Decouples from `bash`, simplifies (#3577 (comment))
dangotbanned added a commit that referenced this issue Dec 29, 2024
@dangotbanned
ci(DRAFT): migrate hatch scripts to hybrid tasks.py runner
6ebc0ee 
Provides 2 interfaces:
- `tasks.py`: to define using decorators
- `tasks.toml`: an example of defining in a similar way to `hatch` scripts

#3577 (comment)
@dangotbanned dangotbanned linked a pull request Dec 29, 2024 that will close this issue
Draft
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement help wanted
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

build(DRAFT): Migrate from hatch to uv vega/altair
4 participants
@joelostblom @jonmmease @binste @dangotbanned