Change filename of wheel with build number based on length of changelog

[shubhbapna]

• Bump the build tag of wheel name based on the length of changelog: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-name-convention
• Start all of the actual builds with 0 to differentiate them from upstream packages (need to test that pip and gitlab pypi can handle this)
• Do not change wheel version metadata
• Make any changes to resolver code and finder code to account for this new wheel file name
  • Resolver in fromager should not look at the build number because it can’t use build number to pick a specific artifact. The caller should look at the wheel filename returned by the resolver

[shubhbapna]

@dhellmann I am not sure if we can implement this without changing the resolver code. I have been playing around with resolvelib and if there are 2 distributions of the same version with different build tag then it takes the one that appears first. Not sure how to enforce it to take the latest one. If we can't get the latest one then the caller won't be able to figure out the latest build tag from the url

I tried passing something like Requirement("foo==1.0.0-1") to the resolver and it doesn't seem to work (the test env does have a foo-1.0.0-1 wheel).

[dhellmann]

@dhellmann I am not sure if we can implement this without changing the resolver code. I have been playing around with resolvelib and if there are 2 distributions of the same version with different build tag then it takes the one that appears first. Not sure how to enforce it to take the latest one. If we can't get the latest one then the caller won't be able to figure out the latest build tag from the url

Is resolvelib picking the file or is that something we're doing in the candidate selection code in fromager?

I'm OK with changing the resolver code to make sure we get the latest file, I just didn't want to force the resolver to have to know how to pick something with a build tag that wasn't the latest just for the optimization case of seeing if we already had that package. We aren't going to rebuild something with an old build tag, we would build it again with a new tag.

[shubhbapna]

Is resolvelib picking the file or is that something we're doing in the candidate selection code in fromager?

It is picking with the help of the candidate selection code in fromager. So the first thing that satisfies the requirements and constraints is picked up

[dhellmann]

Is resolvelib picking the file or is that something we're doing in the candidate selection code in fromager?

It is picking with the help of the candidate selection code in fromager. So the first thing that satisfies the requirements and constraints is picked up

OK, so it sounds like we need to teach our candidate selection code about build numbers.

[shubhbapna]

@tiran @dhellmann how should i go about adding build tags. I am not able to find anything within the pip wheel command (maybe some --global-option or --build-option but I dont know how those work).

I saw another tool called wheel which as a wheel tags command for exactly this purpose. Should I use that?
https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html

Or i can use a regex to reconstruct the wheel filename

[shubhbapna]

I think I found the correct options -C="--build-option=--build-number" -C="--build-option=999
Is this specific to setuptools or can it be used generally?

[tiran]

As far as I know, the option is specific to the build backend. It works for setuptools but it may not work for flit, hatch, or maturin backends.

We should look into build for wheel and sdist building with PEP 517. With build, you can do:

python3 -m build -w -C='--build-option=--build-number=999'

-C passes the build option to the build backend. The build backend passes --build-number to the wheel builder.

Wheel and build even include the build number in the package's dist-infi WHEEL file:

Wheel-Version: 1.0
Generator: setuptools (72.2.0)
Root-Is-Purelib: true
Build: 999
Tag: py3-none-any

References:

• https://build.pypa.io/en/stable/
• https://pip.pypa.io/en/stable/cli/pip_wheel/#cmdoption-build-option
• https://wheel.readthedocs.io/en/stable/reference/wheel_pack.html

[shubhbapna]

We should look into build for wheel

What is the difference between pip wheel and build?

[tiran]

build is a tool just for building sdists and wheels.

The config settings and build options don't work reliable for me. I have another idea:

• put the new wheel into a temporary location
• use wheel unpack to unpack a temporary wheel (and then delete it)
• modify the wheel's dist info with additional metadata: changelog, build environment info, ELF requires, ELF provides, ...
• use build pack --build-number=<tag> to create a new wheel file

This works for any wheel, even external wheels.

[shubhbapna]

If we can use the wheel command then i think using just wheel tags will get us exactly what we want for now.

For changing the metadata I will keep this in mind for #319

[shubhbapna]

Ran a test to see if pip install itself picks up the latest build tag for the same version and it seems like it does!

My local simple repository:

pip command:

(venv) $pip install -i http://localhost:8080/simple stevedore
Looking in indexes: http://localhost:8080/simple
Collecting stevedore
  Downloading http://localhost:8080/simple/stevedore/stevedore-5.2.0-2-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.6/48.6 kB 395.1 MB/s eta 0:00:00
Collecting pbr!=2.1.0,>=2.0.0 (from stevedore)
  Downloading http://localhost:8080/simple/pbr/pbr-6.0.0-0-py2.py3-none-any.whl (105 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.1/105.1 kB 379.4 MB/s eta 0:00:00
Installing collected packages: pbr, stevedore
Successfully installed pbr-6.0.0 stevedore-5.2.0

[notice] A new release of pip is available: 24.1 -> 24.2
[notice] To update, run: pip install --upgrade pip

Contents of WHEEL file:

Wheel-Version: 1.0
Generator: setuptools (71.1.0)
Root-Is-Purelib: true
Tag: py3-none-any
Build: 2

Looks like we can go ahead with building wheels this way

[shubhbapna]

We can move to the public API instead of using the wheel CLI once this lands: pypa/packaging#805

[shubhbapna]

How should we go about handling the case where the user has removed an entry from the changelog and the build tag in the url returned from the resolver is higher than what is expected

Options:

• Raise an error and stop the build
• Add +1 to the build tag gotten from the url and warn the user of an invalid changelog

Probably replicating the rpm behaviour would be the best option (not sure what that is though)

[dhellmann]

I like the idea of just incrementing the number again, but I worry that might lead to unintended side-effects, so let's think about it some more before going with that.