GH-37929: [Python] begin moving static settings to pyproject.toml

[anjakefala]

Rationale for this change

To migrate Arrow to modern Python packaging standards, see PEP-517 and PEP-518.

• GitHub Issue: [Python] Move setuptools-scm configuration to pyproject.toml #37929

This PR focuses on migrating the static settings, the metadata and version, to pyproject.toml. Future PRs will migrate more of the build process to pyproject.toml.

[github-actions]

⚠️ GitHub issue #37929 has been automatically assigned in GitHub to PR creator.

[anjakefala]

I will try to investigate how other projects handle development versions with pyproject.toml!

[anjakefala]

I found this! https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata I'll get to work updating.

[anjakefala]

I am actively trying to figure out how to fix what's coming up in the build failures!

[anjakefala]

This is the current challenge for this PR.

To migrate the static metadata to the pyproject.toml, we need to set a version. In PyArrow, the version is set dynamically using setuptools_scm.

Setuptools_scm will only let you configure Callables in the setup.py. There are two callables we set, one for parse and one for version_scheme.

 429 def parse_git(root, **kwargs):                                                                                                                                                                                                
 430     """                                                                                                                                                                                                                       
 431     Parse function for setuptools_scm that ignores tags for non-C++                                                                                                                                                           
 432     subprojects, e.g. apache-arrow-js-XXX tags.                                                                                                                                                                               
 433     """                                                                                                                                                                                                                       
 434     from setuptools_scm.git import parse                                                                                                                                                                                      
 435     kwargs['describe_command'] =\                                                                                                                                                                                             
 436         'git describe --dirty --tags --long --match "apache-arrow-[0-9]*.*"'                                                                                                                                                  
 437     return parse(root, **kwargs)                                                                                                                                                                                              
 438                                          

 440 def guess_next_dev_version(version):                                                                                                                                                                                          
 441     if version.exact:                                                                                                                                                                                                         
 442         return version.format_with('{tag}')                                                                                                                                                                                   
 443     else:                                                                                                                                                                                                                     
 444         def guess_next_version(tag_version):                                                                                                                                                                                  
 445             return default_version.replace('-SNAPSHOT', '')                                                                                                                                                                   
 446         return version.format_next_version(guess_next_version)                                                                                                                                                                
 447                                                                

As they currently are, we cannot configure these in the pyproject.toml, it will not accept a Python callable.

The next part of the challenge is that if you move the version metadata to pyproject.toml, none of the configurations in setup.py will be picked up. That is why the build is failing. So you cannot put the static variables in pyproject.toml, and pass the Python callables into setup.py via use_scm_version. It is an all-or-nothing migration.

I'm thinking that the next step is to contact the maintainers of setuptools_scm, and see if they have any advice.

[jorisvandenbossche]

The setuptools_scm docs have an example of passing a callable in setup.py with using pyproject.toml: https://setuptools-scm.readthedocs.io/en/latest/customizing/#providing-project-local-version-schemes
So based on that it seems this should be possible?

[jorisvandenbossche]

Also, it seems that we use parse_git callable to use a custom git describe invocation. But, nowadays setuptools_scm also has an option to directly override which describe command is used (git_describe_command). So that part can probably be turned into a static configuration instead of a callable.

[anjakefala]

So based on that it seems this should be possible?

I found open issues for the behaviour I am noticing:

pypa/setuptools-scm#827
pypa/setuptools-scm#1011

[anjakefala]

Locally, the test seems to be behaving decently:

(base) ~/git/arrow/python kef/pyproject $ ls dist 
pyarrow-16.0.0.dev453+g51a3831e4.d20240416.tar.gz

The question is understanding why it is failing in CI.

[pitrou]

Note that we're not married to setuptools_scm. If we find out that something else would work better for us, then we can switch to it.

Found this comparison using a quick search: jwodder/versioningit#46 (comment)

[anjakefala]

setuptools_scm does have logging. I'm going to see if the logging helps reveal anything. If nothing, I'll explore the alternatives!

[raulcd]

I've rebased main to fix conflicts

[raulcd]

@github-actions crossbow submit -g python

[github-actions]

Revision: 50a35c1

Submitted crossbow builds: ursacomputing/crossbow @ actions-77bf8dcbea

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

[jorisvandenbossche]

@github-actions crossbow submit example-python-minimal-build-ubuntu-venv

[github-actions]

Revision: 24054ef

Submitted crossbow builds: ursacomputing/crossbow @ actions-c870c684c4

Task	Status
example-python-minimal-build-ubuntu-venv

[raulcd]

I think this can be merged now, @jorisvandenbossche @pitrou any other concerns?

[jorisvandenbossche]

Yes, I would also say, let's merge this ;)

[anjakefala]

Thanks everyone! This turned out to be such a surprising doozy.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit ad897bb.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.