automatic dependency checking for python 3 modules

[classabbyamp]

Goal: lower the incidence of missing dependencies for python module packages (like #53257), by checking what the upstream has indicated are dependencies and cross-referencing against what's been provided in the template.

Secondary goal: provide some infrastructure for decentralising shlibs tracking (working towards what paper started in #40698)

• xbps: ignore /provides file in xbps-create
  • needed for new provides handling
  • no longer needed
• New package: python3-flit_core-bootstrap-3.9.0
  • dependency of python3-packaging-bootstrap and python3-installer-bootstrap
• New package: python3-installer-bootstrap-0.7.0
  • dependency of python3-installer-bootstrap
• New package: python3-packaging-bootstrap-24.1
  • needed for parse-py-metadata script
• common/environment/build-style/python3-*.sh: add python3-packaging-bootstrap
  • only if verifying python dependencies. needed for python module dependency verification hook
  • without this, it would be prohibitively complex to parse python module dependency notation
• xbps-src: adjust help for -s
  • it's now used for python dependency verification too
• common/hooks/post-install/04-create-xbps-metadata-scripts.sh: handle python bootstrap
  • several bootstrap python3 modules put packages in ${py3_sitelib}-bootstrap to prevent conflicts and keep them out of the normal PYTHONPATH. this allows the pycompile_module hook to work for those modules.
• common/environment/setup-subpkg/subpkg: add variables for python dep verification
  • see Manual.md for usage notes
• common/hooks: use file for managing provides
  • this will allow provides to be generated from things like package contents and saved into the file for package generation
• common/scripts/parse-py-metadata.py: add script to parse python module metadata
  • this script uses python3-packaging-bootstrap to extract information about python modules, including their names and dependencies for use by xbps-src hooks
• common/hooks/pre-pkg/04-generate-provides: generate provides for python modules
• common/hooks/pre-pkg/06-verify-python-deps: add hook
  • compares rdeps and Python module metadata to find missing dependencies for python packages. this uses virtual packages to map python canonical package names to void packages.

Testing the changes

• I tested the changes in this PR: YES (extensive testing still in progress)

Output

strict mode enabled:

strict mode disabled:

Related PRs

• bin/xbps-create: ignore provides file xbps#605
• xlint: add new valid variables leahneukirchen/xtools#331
• common: move handling of rdeps, shlib-provides, shlib-requires files #53401

[icp1994]

Is there no way to incorporate pip check rather than hand rolling the metadata verification?

[classabbyamp]

• I'd rather not have pip always installed
• I don't want to do version compatibility checking
• pip check would require installing depends on the chroot, it seems
• introducing py3: provides has other benefits, like being able to find python modules by canonical name

[ahesford]

First issue: what's the suggested approach to broken builds like libxml2 when they build Python modules but don't use one of the python build styles (or helper; at least a few packages explicitly cannot use the helper because the default environment breaks their builds)? Should we

• Force the template to explicitly skip verification;
• Detect import failures in your dependency parser and just exit without error;
• Force the template to manually include python3-packaging-bootstrap; or
• Some other mechanism?

[classabbyamp]

i think dropping a warning if it can't import packaging (so that python3-packaging-bootstrap can be manually included in the template) is the way to go

[classabbyamp]

fixups add a warning if it can't import packaging (this does not get escalated with -s):

[ahesford]

Notionally, this looks good. I'm running through full py3.13 rebuilds now and see the virtuals as expected. I can pull this patchset along with the final py3.13 merge, unless you see a reason to merge stuff piecemeal ahead of time.

I'm looking to merge py3.13 this weekend or next, and I'd like to get at least two other maintainers to sign off on this structural change before including it in the bump.

[classabbyamp]

if this gets merged, I'd like to split off the xbps backport and merge that before the rest of the python bump to ensure the builders get it for their chroots

[ahesford]

I think you should go ahead with the xbps merge, and ideally have xtools ready this week so we can drop the throwaway commit in this PR.

[classabbyamp]

i have an idea for a way to do this without the xbps change

[ahesford]

ada7d18