Skip to content

Commit

Permalink
chore(docs): document virtual_deps and resolutions
Browse files Browse the repository at this point in the history
Fixes a FIXME
  • Loading branch information
alexeagle committed Nov 22, 2024
1 parent aae7c24 commit 23bb8c3
Show file tree
Hide file tree
Showing 9 changed files with 144 additions and 12 deletions.
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
docs/*.md linguist-generated=true
docs/migrating.md linguist-generated=false
docs/virtual_deps.md linguist-generated=false

# Configuration for 'git archive'
# see https://git-scm.com/docs/git-archive/2.40.0#ATTRIBUTES
Expand Down
2 changes: 1 addition & 1 deletion docs/py_binary.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/py_library.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/py_test.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/venv.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

96 changes: 96 additions & 0 deletions docs/virtual_deps.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Resolution of "virtual" dependencies

rules_py allows external Python dependencies to be specified by name rather than as a label to an installed package, using a concept called "virtual" dependencies.

Virtual dependencies allow the terminal rule (for example, a `py_binary` or `py_test`) to control the version of the package which is used to satisfy the dependency, by providing a mapping from the package name to the label of an installed package that provides it.

This feature allows:
- for individual projects within a monorepo to upgrade their dependencies independently of other projects within the same repository
- overriding a single version of a dependency for a py_binary or py_test
- to test against a range of different versions of dependencies for a single library

Links to design docs are available on the original feature request:
https://github.com/aspect-build/rules_py/issues/213

## Declaring a dependency as virtual

Simply move an element from the `deps` attribute to `virtual_deps`.

For example, instead of getting a specific version of Django from
`deps = ["@pypi_django//:pkg"]` on a `py_library` target,
provide the package name with `virtual_deps = ["django"]`.

> Note that any `py_binary` or `py_test` transitively depending on this `py_library` must be loaded from `aspect_rules_py` rather than `rules_python`, as the latter does not have a feature of resolving the virtual dep.
## Resolving to a package installed by rules_python

Typically, users write one or more `pip_parse` statements in `WORKSPACE` or `pip.parse` in `MODULE.bazel` to read requirements files, and install the referenced packages into an external repository. For example, from the [rules_python docs](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-dependencies-from-pypi):

```
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
hub_name = "my_deps",
python_version = "3.11",
requirements_lock = "//:requirements_lock_3_11.txt",
)
use_repo(pip, "my_deps")
```

rules_python writes a `requirements.bzl` file which provides some symbols to work with the installed packages:

```
load("@my_deps//:requirements.bzl", "all_whl_requirements_by_package", "requirement")
```

These can be used to resolve a virtual dependency. Continuing the Django example above, a binary rule can specify which external repository to resolve to:

```
load("@aspect_rules_py//py:defs.bzl", "resolutions")
py_binary(
name = "manage",
srcs = ["manage.py"],
# Resolve django to the "standard" one from our requirements.txt
resolutions = resolutions.from_requirements(all_whl_requirements_by_package, requirement),
)
```

## Resolving directly to a binary wheel

It's possible to fetch a wheel file directly without using `pip` or any repository rules from `rules_python`, using the Bazel downloader.

`MODULE.bazel`:

```
http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
http_file(
name = "django_4_2_4",
urls = ["https://files.pythonhosted.org/packages/7f/9e/fc6bab255ae10bc57fa2f65646eace3d5405fbb7f5678b90140052d1db0f/Django-4.2.4-py3-none-any.whl"],
sha256 = "860ae6a138a238fc4f22c99b52f3ead982bb4b1aad8c0122bcd8c8a3a02e409d",
downloaded_file_path = "Django-4.2.4-py3-none-any.whl",
)
```

Then in a `BUILD` file, extract it to a directory:

```
load("@aspect_rules_py//py:defs.bzl", "py_binary", "py_unpacked_wheel")
# Extract the downloaded wheel to a directory
py_unpacked_wheel(
name = "django_4_2_4",
src = "@django_4_2_4//file",
)
py_binary(
name = "manage.override_django",
srcs = ["proj/manage.py"],
resolutions = {
# replace the resolution of django with that specific wheel
"django": ":django_4_2_4",
},
deps = [":proj"],
)
```

4 changes: 3 additions & 1 deletion py/private/py_library.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,9 @@ _attrs = dict({
default = [],
),
"resolutions": attr.label_keyed_string_dict(
doc = "FIXME",
doc = """Satisfy a virtual_dep with a mapping from external package name to the label of an installed package that provides it.
See [virtual dependencies](/docs/virtual_deps.md).
""",
),
})

Expand Down
3 changes: 2 additions & 1 deletion py/private/py_venv.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,8 @@ py_venv_rule = rule(
mandatory = False,
),
"resolutions": attr.label_keyed_string_dict(
doc = "FIXME",
doc = """Satisfy a virtual_dep with a mapping from external package name to the label of an installed package that provides it.
See [virtual dependencies](/docs/virtual_deps.md).""",
),
"package_collisions": attr.string(
doc = """The action that should be taken when a symlink collision is encountered when creating the venv.
Expand Down
44 changes: 38 additions & 6 deletions py/tests/virtual/django/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,13 @@ load("//py:defs.bzl", "py_binary", "py_library", "py_unpacked_wheel", "resolutio

django_resolutions = resolutions.from_requirements(all_whl_requirements_by_package, requirement)

py_unpacked_wheel(
name = "django_wheel",
src = "@django_4_2_4//file",
)

compile_pip_requirements(
name = "requirements",
requirements_in = "requirements.in",
requirements_txt = "requirements.txt",
)

# Test fixture: a library with an external dependency
py_library(
name = "proj",
srcs = glob(["proj/**/*.py"]),
Expand All @@ -25,6 +21,9 @@ py_library(
virtual_deps = ["django"],
)

## Use case 1
# Resolve it using the result of a rules_python pip.parse call.
# It will use pip install behind the scenes.
py_binary(
name = "manage",
srcs = ["proj/manage.py"],
Expand All @@ -36,13 +35,46 @@ py_binary(
],
)

## Use case 2
# Use a binary wheel that was downloaded with http_file, bypassing rules_python and its
# pip install repository rules.
py_unpacked_wheel(
name = "django_4_2_4",
src = "@django_4_2_4//file",
)

# bazel run //py/tests/virtual/django:manage.override_django -- --version
# Django Version: 4.2.4
py_binary(
name = "manage.override_django",
srcs = ["proj/manage.py"],
# package_collisions = "warning",
# Install the dependencies that the pip_parse rule defined as defaults...
resolutions = django_resolutions.override({
# ...but replace the resolution of django with a specific wheel fetched by http_file.
"django": "//py/tests/virtual/django:django_wheel",
"django": ":django_4_2_4",
}),
deps = [":proj"],
)

## Use case 3
# It's possible to completely remove a dependency.
# For example, to reduce the size of an image when a transitive dep is known to be unused.
filegroup(
name = "empty",
)

# bazel run //py/tests/virtual/django:manage.remove_django -- --version
# ImportError: Couldn't import Django.
# Are you sure it's installed and available on your PYTHONPATH environment variable?
# Did you forget to activate a virtual environment?
py_binary(
name = "manage.remove_django",
srcs = ["proj/manage.py"],
package_collisions = "warning",
resolutions = django_resolutions.override({
# Replace the resolution of django with an empty folder
"django": ":empty",
}),
deps = [":proj"],
)

0 comments on commit 23bb8c3

Please sign in to comment.