argparse.HelpFormatter subclasses may be broken by 3.14 changes

[iritkatriel]

I came across this trying to run mypy benchmarks. The version I had contained

class AugmentedHelpFormatter(argparse.RawDescriptionHelpFormatter):
    def __init__(self, prog: str) -> None:
        super().__init__(prog=prog, max_help_position=30)

Which was rewritten in a newer version to:

class AugmentedHelpFormatter(argparse.RawDescriptionHelpFormatter):
    def __init__(self, prog: str, **kwargs: Any) -> None:
        super().__init__(prog=prog, max_help_position=30, **kwargs)

The changes in #124456 and #132323 can be made without breaking user code, if they accept the new args as **kwargs, extract their values from kwargs if they are there, and continue working as before if not. Is there a good reason not to do this?

Do these changes currently comply with Python's backwards compatibility policy?

Linked PRs

• gh-133653: Fix subclassing HelpFormatter #133668
• gh-133653: Fix argparse.ArgumentParser with the formatter_class argument #133813
• [3.14] gh-133653: Fix argparse.ArgumentParser with the formatter_class argument (GH-133813) #133941

[iritkatriel]

CC @hugovk @savannahostrowski .

[hugovk]

The changes in #124456 and #132323 can be made without breaking user code, if they accept the new args as **kwargs, extract their values from kwargs if they are there, and continue working as before if not. Is there a good reason not to do this?

Let's do this.

Would you like to open a PR? I'm also happy to do it.

[iritkatriel]

Please go ahead.

[iritkatriel]

Something like

        prefix_chars = kwargs.get('prefix_chars', '-')
        color = kwargs.get('color', False)

[The-Compiler]

Do these changes currently comply with Python's backwards compatibility policy?

FWIW, argparse.HelpFormatter itself is undocumented and the docstrings say:

cpython/Lib/argparse.py

Lines 62 to 64 in 0ec8fc8

	(Also note that HelpFormatter and RawDescriptionHelpFormatter are only
	considered public as object names -- the API of the formatter objects is
	still considered an implementation detail.)

cpython/Lib/argparse.py

Lines 157 to 162 in 0ec8fc8

	class HelpFormatter(object):
	"""Formatter for generating usage messages and argument help strings.
	Only the name of this class is considered a public API. All the methods
	provided by the class are considered an implementation detail.
	"""

But despite that, subclassing argparse.HelpFormatter seems to be a common occurrence, and many implementations seem to not use **kwargs.

[hugovk]

How does PR #133668 look?

[serhiy-storchaka]

I think this is a documentation issue. The only way to fix it reliably is to properly document the HelpFormatter class and how it cab be extended.

Currently the HelpFormatter class is not documented. It's subclassing is not officially supported. You need to look in the source code to find about max_help_position.

[iritkatriel]

The only way to fix it reliably is to properly document the HelpFormatter class and how it cab be extended.

As discussed on the PR - doc change is welcome but it is not the only way to fix this.

I think we should add a setter method for the new fields so that the signature of __init__ doesn't need to change.

We saw two libraries break already, there will be more post release. There is no point repeatedly litigating this with users over the next 3-5 years if we can avoid that with a simple change now.

[serhiy-storchaka]

There is something similar issue #133745. It is more serious, because the documented contract was changed. For HelpFormatter there is no documented contract. We can apply solution similar to the code that was used before #128308 -- pass only prog to the factory, and set other parameters as private HelpFormatter attributes. This approach has many drawbacks. It still can break custom formatters, and for some users it will only defer breakage fr future versions. In long run, it is better to break the user code now. But we need the documentation to make sure that users will write correct code which will not break in future.