Preparing for Python 3.8 EOL

[dangotbanned]

What is your suggestion?

Based on PEP 569 and devguide official support for Python 3.8 will be ending in October 2024. At the time of writing this is 2-3 months away.

The last bump altair had was #3100 which was a few weeks after 3.7 EOL.

I see one small (#3498), and another larger PR to come from this.

Bumping build.requires-python

I'm not proposing doing this immediately, just to keep on the radar and present some findings.

altair/pyproject.toml

Line 35 in f2ac0a1

requires-python = ">=3.8"

Also updating these two will provide autofixes/warnings for what changes are relevant

altair/pyproject.toml

Line 187 in f2ac0a1

target-version = "py38"

altair/pyproject.toml

Line 405 in f2ac0a1

pythonVersion="3.8"

Related Projects

SPEC 0 — Minimum Supported Dependencies

Descending order by requires-python:

requires-python	project	source
3.10	pandas	https://github.com/pandas-dev/pandas/blob/67a58cddc2f8c0e30cb0123589947d9b3f073720/pyproject.toml#L28
3.10	numpy	https://github.com/numpy/numpy/blob/d3d7b55616a062d91af8e20d2a4eb817a41cea56/pyproject.toml#L19
3.10	ibis	https://github.com/ibis-project/ibis/blob/aa1eff703a0b53899b5e70e1a08ac23ee23e2a2c/pyproject.toml#L39
3.10	scipy	https://github.com/scipy/scipy/blob/a1ea3b4a5af724b47cc22160b379db563fbf83ed/pyproject.toml#L47
3.10	ipython	https://github.com/ipython/ipython/blob/ad07901d8cf6fb874262c6387bd24d9e5a284144/pyproject.toml#L23
3.9	geopandas	https://github.com/geopandas/geopandas/blob/2d2d94b1a67e0f1cff7bcd9d2bf4f603c936d027/pyproject.toml#L22
3.8	pyarrow	https://github.com/apache/arrow/blob/a88f0cd0368f81f1b9243d443cd571d09242c5b1/python/pyproject.toml#L37
3.8	polars	https://github.com/pola-rs/polars/blob/69b20646fa8090d76241522f0f8b8b24edcb4ae0/py-polars/pyproject.toml#L13
3.8	jsonschema	https://github.com/python-jsonschema/jsonschema/blob/d2aa2ba124417484eb8ed0950d48527a50b0f773/pyproject.toml#L11

Testing

Currently Python 3.9 is not part of the github test matrix:

altair/.github/workflows/build.yml

Line 10 in f2ac0a1

python-version: ["3.8", "3.10", "3.11", "3.12"]

However I had previously added 3.9 to the hatch test matrix, providing coverage locally with hatch test --all:

altair/pyproject.toml

Lines 139 to 140 in f2ac0a1

	[[tool.hatch.envs.hatch-test.matrix]]
	python = ["3.8", "3.9", "3.10", "3.11", "3.12"]

This one is a pretty minor fix, and could be added now - since eventually it will be the lower bound.

[dangotbanned]

To add some extra detail on SPEC 0 — Minimum Supported Dependencies.

The recommendation there is to drop support for python<=3.10.* in October 2024.

Endorsed By

• https://github.com/ipython/ipython
• https://github.com/numpy/numpy
• https://github.com/scipy/scipy
• https://github.com/scikit-image/scikit-image
• https://github.com/pydata/xarray
• https://github.com/matplotlib/matplotlib
• https://github.com/networkx/networkx
• https://github.com/zarr-developers/zarr-python

Personally think existing altair users that wish to update, will also likely have at least one of these installed.

---

ipython staggered their adoption of SPEC 0 in minor releases:

• python>=3.9
• python>=3.10

So there is precedent for this in a project most likely used with altair

[mattijn]

Last year we were bound to use jsonschema 3 as this was still used in downstream products that are utilised in teaching environments, see eg #3038 (comment). This may have changed now, but I think that the required python version of our minimum supported version of jsonschema will also define our minimum required python version.

[dangotbanned]

Last year we were bound to use jsonschema 3 as this was still used in downstream products that are utilised in teaching environments, see eg #3038 (comment).

Thanks @mattijn for linking #3038, that helps explain some compatibility code in schemapi.

This may have changed now, but I think that the required python version of our minimum supported version of jsonschema will also define our minimum required python version.

On this point, I tried following the linked thread (https://community.deepnote.com/c/ask-anything/jsonschema-4-0-1) but it just redirects to (https://deepnote.com/).

This seems to have been a recent change, as an ipywidget-related issue opengeos/leafmap#751 used links to their forum last month.
However, the resolution for that issue was that deepnote should fix.

---

While I can see argument for their educational access - this is still a proprietary, closed-source product. Maybe this has changed in the past year?

Reading through their docs, the default environment uses python==3.9 and includes these libraries which does not mention altair, jsonschema.

Interestingly, there is an overlap between their requirements and SPEC 0 — Minimum Supported Dependencies with the following libraries:

The recommendation there is to drop support for python<=3.10.* in October 2024.

Endorsed By

• numpy/numpy
• scipy/scipy
• matplotlib/matplotlib

---

#3038 (comment)

I learned in a thread on their forum that "jsonschema is one of the core packages of Deepnote that make the product work".

I found a conda environment section that does include altair, jsonschema.

[MarcoGorelli]

The recommendation there is to drop support for python<=3.10.* in October 2024.

Is there any significant benefit for Altair to drop 3.9? If not, I'd like to suggest that it not be done

Anecdotally, one client I have needed some pandas code optimising with as few API changes as possible, but they were stuck on Python3.8, so many of the pyarrow improvements in pandas were unavailable to them. Polars, on the other hand, still supports 3.8, so a simple solution was to just deliver them some Polars functions wrapped in the pandas API they were familiar with 😎 We could use the latest Polars version for that, no problem

pandas didn't really have a choice about dropping 3.8 because it depends on NumPy. But for Polars, supporting 3.8 until its EOL wasn't much effort, and it's ended up benefitting their adoption.
Unless it's a lot of effort to support 3.9 in Altair, I'd suggest not dropping it here until its EOL

[mattijn]

OK, it seems they are still on jsonschema version 3.2, see opengeos/leafmap#751 (comment). If that works with python 3.9 we might be able to bump too. If not, then I agree with @MarcoGorelli, than just wait.

[dangotbanned]

Anecdotally, one client I have needed some pandas code optimising with as few API changes as possible, but they were stuck on Python3.8, so many of the pyarrow improvements in pandas were unavailable to them. Polars, on the other hand, still supports 3.8, so a simple solution was to just deliver them some Polars functions wrapped in the pandas API they were familiar with 😎 We could use the latest Polars version for that, no problem

pandas didn't really have a choice about dropping 3.8 because it depends on NumPy. But for Polars, supporting 3.8 until its EOL wasn't much effort, and it's ended up benefitting their adoption. Unless it's a lot of effort to support 3.9 in Altair, I'd suggest not dropping it here until its EOL

Thanks for the insight @MarcoGorelli

Is there any significant benefit for Altair to drop 3.9? If not, I'd like to suggest that it not be done

Generally speaking, I've been viewing this as an opportunity to simplify based on upstream consensus around a standard - but you do make a good point on adoption.

My perspective may be skewed, since I'm writing if sys.version_info >= ... more frequently for typing - when compared to most other altair code.

For the stdlib, these are the features that I'm interested in/seen compat code for in altair

Python 3.9-3.12 relevant features

3.9

• dictionary-merge-update-operators
• new-string-methods-to-remove-prefixes-and-suffixes
• graphlib
• typing
  • Runtime type-hinting-generics-in-standard-collections
  • deprecated-aliases

3.10

• pep-634-structural-pattern-matching
• importlib-metadata
• typing
  • Runtime pep-604-new-type-union-operator
  • pep-612-parameter-specification-variables

3.11

• pep-678-exceptions-can-be-enriched-with-notes
• typing
  • pep-655-marking-individual-typeddict-items-as-required-or-not-required
  • pep-673-self-type

3.12

• pep-701-syntactic-formalization-of-f-strings
• typing
  • Runtime pep-695-type-parameter-syntax
  • pep-692-using-typeddict-for-more-precise-kwargs-typing
  • pep-698-override-decorator-for-static-typing

The recommendation there is to drop support for python<=3.10.* in October 2024.

Based on those alone, my opinion would be that:

• 3.8 -> 3.9 - worth it and expected
• 3.9 -> 3.10 - would be nice to have but not as big of a change
• 3.10 -> 3.11 - not recommending
• 3.12 - too recent, but has more benefits than previous

[binste]

I could imagine that the advantages are larger for other more complex packages in the scientific ecosystem. For Altair, I'd also prefer if we have faster CI pipelines with less versions to test and less compatibility code but it still feels manageable to me.

Thanks for compiling the features of interest for each version, this allows for a much more informed decision! Based on this, I'd also be in favour of continuing to drop support for Python versions when they reach EOL and support the remaining ones, i.e. drop support for 3.8 in the first release after October 2024.

Also, there is always the chance that we exclude a user group of which we are not aware off (in addition to Deepnote users). Earlier this week, pypa/setuptools#4519 was a good example that tis can happen :)

[dangotbanned]

Seen a few mentions of SPECs recently (1, 2, 3, 4)

I think the prior discusssion here was good - not looking re-discuss adopting SPEC 0.
However, I do think it would be helpful to itemize all potential blockers to adoption

[jonmmease]

I would also encourage us to keep Python 3.9 support until its EOL. A strength of Vega-Altair is its light dependency tree and broad compatibility with fairly old versions of it's optional dependencies. As a user, it's great to be able to use the latest version of Altair in a large complex environment that's stuck on older versions of Python, pandas, numpy, pyarrow, etc.

[binste]

+1. At the NumFOCUS summit we talked to some of the authors of SPEC 0 and they confirmed that it's formulated too strict and they might want to adapt the wording. The original intention was that this is a guidance so you "don't feel guilty" when dropping support for older versions but if you can support more with a reasonable effort, that's great.

[dangotbanned]

Seen a few mentions of SPECs recently (1, 2, 3, 4)

I think the prior discusssion here was good - not looking re-discuss adopting SPEC 0. However, I do think it would be helpful to itemize all potential blockers to adoption

@binste @jonmmease I think my comment should have been clearer.

I'm happy with the decision to not adopt SPEC 0, especially after looking more closely at how altair might benefit.

My aim was to collect more concrete examples like those mentioned earlier in the thread:

• "'ValidationError' object has no attribute 'json_path'" on interactive legend example in RC  #3038 (comment)
• leafmap on deepnote causes exception: "field() got an unexpected keyword argument 'alias'" opengeos/leafmap#751 (comment)

I thought it could be useful to refer to in the future.
For example, if the topic was brought up in 1-3 years and the landscape had changed

[mattijn]

Technically you can still endorse SPEC-0 without strictly abiding to it..