Dropping Python 3.7: why not?

[nolar]

Python 3.7 support is considered to be dropped, restricting Kopf to Python >= 3.8.

Why not >= 3.9? One reason considered in #74 (for Python 3.6) was that some companies restrict their environments to LTS versions only (vendor-provided, or for any other reasons), so it is too early for such a framework-level requirement (the operators can use any versions of Python).

The latest Ubuntu LTS (20.04 since Apr'2020) contains Python 3.8 by default. And it was out there for 8 months already — enough time to upgrade and migrate.

RedHat Enterprise Linux 8 (released May'2019) commits to Python 3.6 support for the whole 8.x series, but RHEL 8.2 (Apr'2020) officially provides Python 3.8 in addition to 3.6 (link).

So, there seems to be no reason to support Python 3.7 even for these restrictive use-cases.

QUESTIONS:

• Can Python 3.7 support be dropped from Kopf safely?
• How will it affect you and your operators?
• Do you have a low-effort upgrade scenario in place?
• Any other considerations?

Tell us why this might be a bad idea to drop Python 3.7 support.

[MarkusH]

AFAICT, Kopf, as a framework, will only be used to build operators. They then run as some form of container "inside" Kubernetes. The operators will (unlikely?) run on some server connecting into K8s "from the outside". So, from where I'm sitting, requiring new Python versions isn't a matter of LTS support on production hosts, but rather availability of the Python version on developer machines.

[nolar]

Asked here: https://twitter.com/nolar/status/1344953069361115142 — some answers give a hint:

Long story short, it seems that most Kubernetes-accessing apps (those using the official client, not just running inside of Kubernetes) are stuck in Python 3.7, barely upgrade even to 3.8. And, among all packages (really all), it also looks like Python 3.7 is growing its share despite 3.8 is there for months-years.

The conclusion is: it makes sense to support 3.7 for some time (a couple of years?) before reconsidering.

---

PyPI shares its raw data which can be analysed with SQL: https://packaging.python.org/guides/analyzing-pypi-package-downloads/

Here are some anchor libraries to see the state of things and the trends (made on 01.01.2021):

The SQL query used:

-- Analyse PyPI package usage by month & version in a columnar layout.
SELECT
  m.month AS `month`,
  py36.cnt as `py36`,
  py37.cnt as `py37`,
  py38.cnt as `py38`,
  py39.cnt as `py39`,
  py310.cnt as `py310`
FROM (
  SELECT DATE_TRUNC(DATE(timestamp), MONTH) AS `month`
  FROM `the-psf.pypi.file_downloads`
  WHERE file.project = 'kopf'
    AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY) AND CURRENT_DATE()
  GROUP BY `month`
) AS M
LEFT JOIN (
  SELECT DATE_TRUNC(DATE(timestamp), MONTH) AS `month`, COUNT(*) AS `cnt`
  FROM `the-psf.pypi.file_downloads`
  WHERE file.project = 'kopf'
    AND details.python LIKE '3.6%'
    AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY) AND CURRENT_DATE()
  GROUP BY `month`
) AS py36 ON py36.month = m.month
LEFT JOIN (
  SELECT DATE_TRUNC(DATE(timestamp), MONTH) AS `month`, COUNT(*) AS `cnt`
  FROM `the-psf.pypi.file_downloads`
  WHERE file.project = 'kopf'
    AND details.python LIKE '3.7%'
    AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY) AND CURRENT_DATE()
  GROUP BY `month`
) AS py37 ON py37.month = m.month
LEFT JOIN (
  SELECT DATE_TRUNC(DATE(timestamp), MONTH) AS `month`, COUNT(*) AS `cnt`
  FROM `the-psf.pypi.file_downloads`
  WHERE file.project = 'kopf'
    AND details.python LIKE '3.8%'
    AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY) AND CURRENT_DATE()
  GROUP BY `month`
) AS py38 ON py38.month = m.month
LEFT JOIN (
  SELECT DATE_TRUNC(DATE(timestamp), MONTH) AS `month`, COUNT(*) AS `cnt`
  FROM `the-psf.pypi.file_downloads`
  WHERE file.project = 'kopf'
    AND details.python LIKE '3.9%'
    AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY) AND CURRENT_DATE()
  GROUP BY `month`
) AS py39 ON py39.month = m.month
LEFT JOIN (
  SELECT DATE_TRUNC(DATE(timestamp), MONTH) AS `month`, COUNT(*) AS `cnt`
  FROM `the-psf.pypi.file_downloads`
  WHERE file.project = 'kopf'
    AND details.python LIKE '3.10%'
    AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY) AND CURRENT_DATE()
  GROUP BY `month`
) AS py310 ON py310.month = m.month
ORDER BY m.month

---

The same source, another view:

• All packages
• Kopf
• Kubernetes

It's even worse for Kubernetes-accessing anchors: 3.7 is only slightly above 3.6 (Kopf does not work in 3.6 at all), while 3.8 is far below.

---

This graph of pypi-tools shows that 3.8 is gaining traction, but 3.7 is still expanding.

[nolar]

An updated graph as of 2022-03 incl — someone uses Python 3.7 even more than before. I wonder who and why.

[nolar]

Another reason to not drop Python 3.7 support: PyPy is compatible with Python 2.7, 3.6, and 3.7 only — 3.8+ versions are not mentioned. Kopf runs quite well with PyPy — even with the kopf[full-auth,dev] extras. Dropping 3.7 support means dropping PyPy support, which is unwanted.

[guhcampos]

This was going to be my argument.

We leverage pypy heavily on our architecture with substantial gains but we haven't tested it with kopf yet at scale, so not sure if our production environment will run with cpython or pypy, but all-in-all we developed the good practice of keeping pretty much everything we do compatible with pypy, because it leaves a very low-cost way of trying to extract extra performance of an application with zero-extra code.

For the little I know about kopf architecture, pypy should issue huge performance benefits, as we're pretty much running the same loops/callbacks over and over again with very similar data structures.

[nolar]

PyPy now supports Python 3.8 & 3.9, and Kopf passes these tests with no problem (it is now included in the CI). So, at least the PyPy argument can be now removed.