Persistent dynamic core metadata breaks user assumptions

[hroncok]

Hello,

in Fedora, @frenzymadness wanted to update hatchling from 1.21 to 1.22.2. However, we encountered several Fedora packages using hatchling to regress.

For many of the packages, we essentially do:

• download sdist archive from PyPI
• extract/unpack it
• apply downstream changes (e.g. patches or sed calls to relax an overly specific dependency)
• build a wheel via PEP 517 API

However, the downstream changes to metadata in pyproject.toml are now silently ignored because all the metadata is read from PKG-INFO.

I have looked up the changelog which says:

Metadata for the wheel target now defaults to the PKG-INFO metadata within source distributions

It seems like this has happened in #1309

I am aware of no prior discussion and there is no rationale provided in the PR -- it looks like the change appeared out of the blue. What problem does this solve? What is the purpose of this behavior?

I assume that when I change the metadata in pyproject.toml it will affect the wheel. And many other folks in Fedora assume the same. This change breaks that assumption.

Essentially, this seems like a cache that fails to be invalidated. Would you please consider our use case and try to mitigate the impact of this change by doing for example one of the following?

• check the mtime stamps of pyproject.toml and PKG-INFO -- invalidate PKG-INFO if pyproject.toml is newer
• include a checksum or another indicator in PKG-INFO to see if it is up-to-date (I don't know if the specification for PKG-INFO allows custom headers) and ignore/regenerate it if it is out of sync
• only read metadata from PKG-INFO if working with archived (i.e. not extracted) sdists (I don't know if hatchling can detect that)

Thanks.

[ofek]

This is a standard now: https://peps.python.org/pep-0643/

Are you not encountering this with new releases of for example poetry-core?

[ofek]

For context, the main reason for that PEP is so resolvers can use metadata from source distributions directly without having to go through the build process.

[hroncok]

Have you perhaps linked a wrong PEP? https://peps.python.org/pep-0643/ does not mention PKG-INFO at all.

[hroncok]

Are you not encountering this with new releases of for example poetry-core?

Not that I am aware of. Neither I can see it in their changelog https://github.com/python-poetry/poetry-core/blob/main/CHANGELOG.md or in the code when I search for PKG-INFO https://github.com/search?q=repo%3Apython-poetry%2Fpoetry-core%20PKG-INFO&type=code

[ofek]

That file is documented in the text on top here: https://packaging.python.org/en/latest/specifications/source-distribution-format/

I could be wrong, but I think other backends are simply not following the spec. Flit I think (if it supports 2.2) would be fine because nothing there is dynamic but Poetry and others allow for certain levels of non-standard dynamism similar to Hatchling build hooks.

[ofek]

I'm asking for more feedback! https://discuss.python.org/t/respecting-core-metadata-2-2-when-building-from-source-distributions/48886

[hroncok]

My understanding is that the file exist so sdist can be queried for metadata statically. Not that build backends should use it when generating wheels.

[pfmoore]

A simple fix is surely just to delete the PKG-INFO file?

The larger issue is that by doing this you're essentially breaking the contract of static metadata, which is that any metadata declared in the sdist as static MUST be identical in any wheel built from that sdist - item 1 in the specification here:

When found in the metadata of a source distribution, the following rules apply:

1. If a field is not marked as Dynamic, then the value of the field in any wheel built from the sdist MUST match the value in the sdist. If the field is not in the sdist, and not marked as Dynamic, then it MUST NOT be present in the wheel.
2. If a field is marked as Dynamic, it may contain any valid value in a wheel built from the sdist (including not being present at all).

You have a loophole in that you're not technically building a wheel from the sdist if you're modifying the code in the sdist, but unless you also change the package name or version, you're violating the spirit of the specification (IMO as author of that spec).

[eli-schwartz]

If you are hand-modifying the extracted source code originally pulled out of an sdist, I think it is exceedingly fair to say that your hand modifications preclude you being the target userbase of:

As a result, metadata consumers are unable to rely on the data available from source distributions, and need to use the (costly) PEP 517 build mechanisms to extract medatata.

I also think that a PEP 517 build backend, itself, doesn't have a good argument to make that they are unable to rely on the data available from source distributions "and need to use the costly PEP 517 build mechanism to extract it".

That being said it also seems to me the spec mandates that hatchling doesn't have a permitted option other than to parse both the dynamically generated metadata as well as the static PKG-INFO metadata, compare them and require them to match, then fatally error out if they don't match.

So people patching this out would still need to delete PKG-INFO. However, at least they would know they have to do so since the build would error out rather than silently generating the wrong contents.

Emitting a fatal error on mismatched contents but using the pyproject.toml as the canonical version would have also caught #1325.

[pfmoore]

That being said it also seems to me the spec mandates that hatchling doesn't have a permitted option other than to parse both the dynamically generated metadata as well as the static PKG-INFO metadata, compare them and require them to match, then fatally error out if they don't match.

The spec doesn't mandate that backends recalculate the values and compare, although I can understand the argument that it's a reasonable check to make to catch corrupt sdists. Having said that, though, I can also understand the argument that manually editing a sdist is unsupported (just like manually editing a wheel would be - both are built distributions) and therefore there's no need to make a potentially costly and error-prone¹ check.

Footnotes

1. Calculating the version metadata from VCS details like tag names will fail because a sdist isn't a valid VCS checkout, for example. ↩

[eli-schwartz]

Calculating the version metadata from VCS details should, in any well-designed plugin -- I'll confess I haven't looked around at most of them but I do know how setuptools-scm works and that's basically the industry standard anyway (hatch-vcs is a wrapper on top of it, for example) -- anyways be using PKG-INFO as the primary source of truth before running git.

And have done so since before metadata 2.2, since that's fundamentally "the" way to know that information if there is no git repo, as there will not be in an sdist.

[hroncok]

A simple fix is surely just to delete the PKG-INFO file?

Yes. It's just not what people expect to need to do.

Either way, I think the real confusion here is based on the fact that the source distributions are now treated as built distributions and if we reuse them downstream as sources we are probably misusing them. While I see where this is coming from, it is going to be particularly hard to explain to people.

[ofek]

Just to be clear it's perfectly acceptable (necessary in many cases) for the wheel to have dynamic contents; it's only the metadata that is static.

[hroncok]

There is a lot of talk about dynamic vs. static metadata. I know that metadata in the [project] table is either listed statically or listed in the dynamic key. I don't know how that's relevant to PKG-INFO.

I see I used the term "dynamic core metadata" in the issue title and perhaps I was misled to do that by the commit message of #1309. In fact, the issue we have is not relevant to whether the data is statically listed in the [project] table or listed in its dynamic key.

Perhaps I should have said Persistent core metadata breaks user assumptions instead. Should I edit the issue?

[eli-schwartz]

I don't think that is a point of confusion.

It is standard practice in the world of software vendors to apply patches to the software they distribute as required in order to ensure the software is correctly integrated for distribution. There is no arbitrary restriction on a software vendor that they can apply patches to a .py file but cannot apply patches to the metadata.

Fedora uses the wheel dist-info metadata to generate rpm dependencies. They need the dist-info metadata to be maximally correct in order for the built rpm to be maximally correct (another option would be to give up on generating rpm dependencies from the dist-info).

That means that if it is necessary to prevent users from updating their rpms of one package to something too new for a second package, for example: because the second package was published without max version pinning but after it was released its dependency (the first package, which we shall say doesn't follow semver) released a new version with a breaking change...

... the canonical rpm way to do this is to patch out pyproject.toml, and let that propagate into the rpm.

This is a serious problem for many PyPI packages, and a point that conda advocates use in favor of conda: it is possible to patch the metadata of a conda package without re-releasing a new package. Many people have wished for the ability to edit old PyPI releases to add upper pins for known after-the-fact discovered incompatibilities.

I believe the point @hroncok is trying to make is that this effectively means that Fedora has to stop advising people to use pypi.org sdists and start using github autogenerated archives instead, since "sdists are built distributions and cannot be modified, but only sometimes, it depends on what you want to modify" is difficult to effectively communicate to people whereas "do not use pypi.org" is easy.

...

If there's an argument to be made that:

unless you also change the package name or version, you're violating the spirit of the specification

holds true when downloading an sdist, but not when git cloning the repo or downloading a github autogenerated archive, then doing the latter is the only spec-valid option.

And if the promise of static metadata is bound to the package name + version, then git cloning the repo or downloading a github autogenerated archive should also be a spec violation, which would leave us at "it is a violation of the Metadata 2.2 spec to distribute third-party packages that have metadata which is in disagreement with the metadata that PyPI registers for the same name+version", in other words, it's a Metadata 2.2 spec violation to exercise one's Open Source Software right to modify and redistribute software.

So I dunno what the answer here is. :) Other than that I wish tools wouldn't allow errors to happen silently.