Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "Modernizing Superseded Typing Features" guide #1541

Merged
merged 6 commits into from
Jan 8, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/guides.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,6 @@ Type System Guides

libraries
writing_stubs
modernizing
unreachable
typing_anti_pitch
336 changes: 336 additions & 0 deletions docs/source/modernizing.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,336 @@
.. role:: python(code)
:language: python

.. role:: t-ext(class)

.. _modernizing:

**************************************
Modernizing Superseded Typing Features
**************************************

Introduction
============

This guide helps to modernize your code by replacing older typing features
with their modern equivalents. Not all features described here are obsolete,
but they superseded by more modern alternatives, that are recommended to use.
srittau marked this conversation as resolved.
Show resolved Hide resolved

These newer features are not available in all Python versions, although
some features are available as backports from the
`typing-extensions <https://pypi.org/project/typing-extensions/>`_
package, or require quoting or using :python:`from __future__ import annotations`.
Each section states the minimum Python version required to use the
feature, whether it is available in typing-extensions, and whether it is
available using quoting.

.. note::

The latest version of typing-extensions is available for all Python
versions that have not reached their end of life, but not necessarily for
older versions.

.. note::

:python:`from __future__ import annotations` is available since Python 3.7.
This will only work inside type annotations, while quoting is still
srittau marked this conversation as resolved.
Show resolved Hide resolved
required outside. For example, this example runs on Python 3.7 and up,
although the pipe operator was only introduced in Python 3.10::

from __future__ import annotations
from typing_extensions import TypeAlias

def f(x: int | None) -> int | str: ... # the future import is sufficient
Alias: TypeAlias = "int | str" # this requires quoting

.. _modernizing-type-comments:

Type Comments
=============

*Alternative available since:* Python 3.0, 3.6

Type comments were originally introduced to support type annotations in
Python 2 and variable annotations before Python 3.6. While most type checkers
still support them, they are considered obsolete, and type checkers are
not required to support them.

For example, replace::

x = 3 # type: int
def f(x, y): # type: (int, int) -> int
return x + y

with::

x: int = 3
def f(x: int, y: int) -> int:
return x + y

When using forward references or types only available during type checking,
it's necessary to either use :python:`from __future__ import annotations`
(available since Python 3.7) or to quote the type::

def f(x: "Parrot") -> int: ...

class Parrot: ...

.. _modernizing-typing-text:

``typing.Text``
===============

*Alternative available since:* Python 3.0

:class:`typing.Text` was a type alias intended for Python 2 compatibility.
It is equivalent to :class:`str` and should be replaced with it.
For example, replace::

from typing import Text

def f(x: Text) -> Text: ...

with::

def f(x: str) -> str: ...

.. _modernizing-typed-dict:

``typing.TypedDict`` Legacy Forms
=================================

*Alternative available since:* Python 3.6

:class:`TypedDict <typing.TypedDict>` supports two legacy forms for
srittau marked this conversation as resolved.
Show resolved Hide resolved
supporting Python versions that don't support variable annotations.
Replace these two variants::

from typing import TypedDict

FlyingSaucer = TypedDict("FlyingSaucer", {"x": int, "y": str})
FlyingSaucer = TypedDict("FlyingSaucer", x=int, y=str)

with::

class FlyingSaucer(TypedDict):
x: int
y: str

But the dictionary form is still necessary if the keys are not valid Python
identifiers::

Airspeeds = TypedDict("Airspeeds", {"unladen-swallow": int})

.. _modernizing-generics:

Generics in the ``typing`` Module
=================================

*Alternative available since:* Python 3.0 (quoted), Python 3.9 (unquoted)

Originally, the :mod:`typing` module provided aliases for built-in types that
accepted type parameters. Since Python 3.9, these aliases are no longer
necessary, and can be replaced with the built-in types. For example,
replace::

from typing import Dict, List

def f(x: List[int]) -> Dict[str, int]: ...

with::

def f(x: list[int]) -> dict[str, int]: ...

This affects the following types:

* :class:`typing.Dict` (→ :class:`dict`)
* :class:`typing.FrozenSet` (→ :class:`frozenset`)
* :class:`typing.List` (→ :class:`list`)
* :class:`typing.Set` (→ :class:`set`)
* :data:`typing.Tuple` (→ :class:`tuple`)

The :mod:`typing` module also provided aliases for certain standard library
types that accepted type parameters. Since Python 3.9, these aliases are no
longer necessary, and can be replaced with the proper types. For example,
replace::

from typing import DefaultDict, Pattern

def f(x: Pattern[str]) -> DefaultDict[str, int]: ...

with::

from collections import defaultdict
from re import Pattern

def f(x: Pattern[str]) -> defaultdict[str, int]: ...

This affects the following types:

* :class:`typing.Deque` (→ :class:`collections.deque`)
* :class:`typing.DefaultDict` (→ :class:`collections.defaultdict`)
* :class:`typing.OrderedDict` (→ :class:`collections.OrderedDict`)
* :class:`typing.Counter` (→ :class:`collections.Counter`)
* :class:`typing.ChainMap` (→ :class:`collections.ChainMap`)
* :class:`typing.Awaitable` (→ :class:`collections.abc.Awaitable`)
* :class:`typing.Coroutine` (→ :class:`collections.abc.Coroutine`)
* :class:`typing.AsyncIterable` (→ :class:`collections.abc.AsyncIterable`)
* :class:`typing.AsyncIterator` (→ :class:`collections.abc.AsyncIterator`)
* :class:`typing.AsyncGenerator` (→ :class:`collections.abc.AsyncGenerator`)
* :class:`typing.Iterable` (→ :class:`collections.abc.Iterable`)
* :class:`typing.Iterator` (→ :class:`collections.abc.Iterator`)
* :class:`typing.Generator` (→ :class:`collections.abc.Generator`)
* :class:`typing.Reversible` (→ :class:`collections.abc.Reversible`)
* :class:`typing.Container` (→ :class:`collections.abc.Container`)
* :class:`typing.Collection` (→ :class:`collections.abc.Collection`)
* :data:`typing.Callable` (→ :class:`collections.abc.Callable`)
* :class:`typing.AbstractSet` (→ :class:`collections.abc.Set`)
srittau marked this conversation as resolved.
Show resolved Hide resolved
* :class:`typing.MutableSet` (→ :class:`collections.abc.MutableSet`)
* :class:`typing.Mapping` (→ :class:`collections.abc.Mapping`)
* :class:`typing.MutableMapping` (→ :class:`collections.abc.MutableMapping`)
* :class:`typing.Sequence` (→ :class:`collections.abc.Sequence`)
* :class:`typing.MutableSequence` (→ :class:`collections.abc.MutableSequence`)
* :class:`typing.ByteString` (→ :class:`collections.abc.ByteString`), but see :ref:`modernizing-byte-string`
* :class:`typing.MappingView` (→ :class:`collections.abc.MappingView`)
* :class:`typing.KeysView` (→ :class:`collections.abc.KeysView`)
* :class:`typing.ItemsView` (→ :class:`collections.abc.ItemsView`)
* :class:`typing.ValuesView` (→ :class:`collections.abc.ValuesView`)
* :class:`typing.ContextManager` (→ :class:`contextlib.AbstractContextManager`)
* :class:`typing.AsyncContextManager` (→ :class:`contextlib.AbstractAsyncContextManager`)
srittau marked this conversation as resolved.
Show resolved Hide resolved
* :class:`typing.Pattern` (→ :class:`re.Pattern`)
* :class:`typing.Match` (→ :class:`re.Match`)

.. _modernizing-union:

``typing.Union`` and ``typing.Optional``
========================================

*Alternative available since:* Python 3.0 (quoted), Python 3.10 (unquoted)

While :data:`Union <typing.Union>` and :data:`Optional <typing.Optional>` are
not considered obsolete, using the ``|`` (pipe) operator is often more
readable. :python:`Union[X, Y]` is equivalent to :python:`X | Y`, while
:python:`Optional[X]` is equivalent to :python:`X | None`.

For example, replace::

from typing import Optional, Union

def f(x: Optional[int]) -> Union[int, str]: ...

with::

def f(x: int | None) -> int | str: ...

.. _modernizing-no-return:

``typing.NoReturn``
===================

*Alternative available since:* Python 3.11, typing-extensions

Python 3.11 introduced :data:`typing.Never` as an alias to
:data:`typing.NoReturn` for use in annotations that are not
return types. For example, replace::

from typing import NoReturn

def f(x: int, y: NoReturn) -> None: ...

with::

from typing import Never # or typing_extensions.Never

def f(x: int, y: Never) -> None: ...

But keep ``NoReturn`` for return types::

from typing import NoReturn

def f(x: int) -> NoReturn: ...

.. _modernizing-type-aliases:

Type Aliases
============

*Alternative available since:* Python 3.12 (keyword); Python 3.10, typing-extensions

Originally, type aliases were defined using a simple assignment::

IntList = list[int]

Python 3.12 introduced the :keyword:`type` keyword to define type aliases::

type IntList = list[int]

Code supporting older Python versions should use
:data:`TypeAlias <typing.TypeAlias>`, introduced in Python 3.10, but also
available in typing-extensions, instead::

from typing import TypeAlias # or typing_extensions.TypeAlias

IntList: TypeAlias = list[int]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might want to also mention typing_extensions.TypeAliasType, which allows providing a name and explicit typevar order. That’s newer and not all checkers support it yet though.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TeamSpen210 As I'm not terribly familiar with TypeAliasType and the new type syntax (yet), could you make a suggestion how we can word this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
IntList: TypeAlias = list[int]
IntList: TypeAlias = list[int]
There are cases where `TypeAlias` is insufficient to express what you can express
using :keyword:`type`, such as when you want the order of type parameters to
be different from how they appear in the body of the alias.
In this case you can use `typing_extensions.TypeAliasType` to express this alias
in older versions of Python::
from typing_extensions import TypeAliasType, TypeVar
T = TypeVar("T", infer_variance=True)
ListOrTuple = TypeAliasType("ListOrTuple", list[T] | tuple[T, ...], type_params=(T, ))
This is equivalent to::
type ListOrTuple[T] = list[T] | tuple[T, ...]
Note however that `TypeAliasType` is newer and not yet supported by every type checker.

Maybe something like this? I couldn't think of a good example where you'd actually want to change the order of type parameters, but it at least conveys how to use TypeAliasType to be equivalent with a type keyword.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I like your text, but I'd like to see an example where this could be useful. I want this document to be a practical guide, so keeping it concise is one of the goals.

Copy link
Contributor

@Daverball Daverball Dec 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair, until mypy supports TypeAliasType there might not be much of a point.

I think there's a potential use-case for using an unbound TypeVar as a slightly more strict gradual type compared to Any. With the old TypeAlias it's not possible to have unbound type params unless you pass them back in explicitly, so that could simplify some code, but pyright does not appear to allow unbound type params inside PEP695 type aliases and just treats it as a type parameter anyways if you ignore the error, so you still have to do it the old way and always pass in the type parameter into the alias to leave it unbound.

To illustrate what I mean:

from collections.abc import Callable
from typing import TypeVar

class Foo:
    pass

class Bar(Foo):
    pass

FooT = TypeVar("FooT", infer_variance=True, bound=Foo)

type AcceptsFoo = Callable[[FooT], object]

# I don't have to specify a type param here, because `AcceptsFoo` takes none
def foo(x: AcceptsFoo) -> None:
    return None
    
def accepts_foo(x: Foo) -> None:
    return None
    
def accepts_bar(x: Bar) -> None:
    return None

foo(accepts_foo)
# This would be rejected if FooT was actually bound
# since FooT should be contravariant, but since we
# leave it unbound it behaves like a gradual type
foo(accepts_bar)
# but this will be rejected since `int` is not a subclass of `Foo`
foo(int)

I think this is a better use-case than the explicit order of type params, but since there's no type checker support for it yet, I can't really use it as a motivating example.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another reason to use TypeAliasType is for at runtime. Since it's an instance of that type and not just an assignment, using it in type annotations will preserve the alias when introspected.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...but since there's no type checker support for [using an unbound type variable] yet...

Using a type variable that is not bound to a scope is expressly forbidden (both in PEP 484 and in PEP 695), so this will never be supported.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erictraut That's fair, but if you modify the above example to this:

from collections.abc import Callable
from typing import TypeVar

class Foo:
    pass

class Bar(Foo):
    pass

FooT = TypeVar("FooT", infer_variance=True, bound=Foo)

type AcceptsFoo[FooT] = Callable[[FooT], object]

def foo(x: AcceptsFoo[FooT]) -> None:
    return None
    
def accepts_foo(x: Foo) -> None:
    return None
    
def accepts_bar(x: Bar) -> None:
    return None

foo(accepts_foo)
foo(accepts_bar)
foo(int)

It will work without raising any errors (beyond the last line) and will work in the way I described it, even though FooT remains unbound in foo, since it only appears in the function signature once.

To me both of those cases should be semantically equivalent. Using a singular TypeVar in functions is a common trick to achieve a gradual type like Any (i.e. it will match in either direction) but that will reject types that don't match the upper bound. But this trick tends to create quite verbose function signatures, which could be avoided if the above were allowed.


.. _modernizing-user-generics:

User Defined Generics
=====================

*Alternative available since:* Python 3.12

Python 3.12 introduced new syntax for defining generic classes. Previously,
generic classes had to derive from :class:`typing.Generic` (or another
generic class) and defined the type variable using :class:`typing.TypeVar`.
For example::

from typing import Generic, TypeVar

T = TypeVar("T")

class Brian(Generic[T]): ...
class Reg(int, Generic[T]): ...

Starting with Python 3.12, the type variable doesn't need to be declared
using ``TypeVar``, and instead of deriving the class from ``Generic``, the
following syntax can be used::

class Brian[T]: ...
class Reg[T](int): ...

.. _modernizing-byte-string:

``typing.ByteString``
=====================

*Alternative available since:* Python 3.0; Python 3.12, typing-extensions

:class:`ByteString <typing.ByteString>` was originally intended to be a type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What exactly this meant is very unclear. This text is probably fine though.

alias for "byte-like" types, i.e. :class:`bytes`, :class:`bytearray`, and
:class:`memoryview`. In practice, this
is seldom exactly what is needed. Use one of these alternatives instead:

* Just :class:`bytes` is often sufficient, especially when not declaring
a public API.
* For items that accept any type that supports the
:ref:`buffer protocol <bufferobjects>`, use :class:`collections.abc.Buffer`
(available since Python 3.12) or :t-ext:`typing_extensions.Buffer`.
* Otherwise, use a union of :class:`bytes`, :class:`bytearray`,
:class:`memoryview`, and/or any other types that are accepted.

``typing.Hashable`` and ``typing.Sized``
========================================

*Alternative available since:* Python 3.12, typing-extensions

The following abstract base classes from :mod:`typing` were added to
:mod:`collections.abc` in Python 3.12:

* :class:`typing.Hashable` (→ :class:`collections.abc.Hashable`)
* :class:`typing.Sized` (→ :class:`collections.abc.Sized`)

Update your imports to use the new locations::

from collections.abc import Hashable, Sized

def f(x: Hashable) -> Sized: ...
Loading