Inferring Generic parameters from function parameters defaults

[dvarrazzo]

This question comes from psycopg/psycopg#308 and from the outdated #1197, and is more focused on only one of the aspects previously asked about (the other is in #1555).

As described in #1555, there is this database adapter (class Connection(Generic[Row])) returning, by default, tuples, but which can be parametrized to return other objects. The attempt in #1555 is based on overloading. Another option would be to exploit a default value for row_factory. What I would expect to be possible is:

from typing import Any, Callable, Generic, Sequence, TypeAlias, TypeVar
from typing_extensions import Self

TupleRow: TypeAlias = tuple[Any, ...]

def tuple_row(values: Sequence[Any]) -> TupleRow:
    raise NotImplementedError


DictRow: TypeAlias = dict[str, Any]

def dict_row(values: Sequence[Any]) -> DictRow:
    raise NotImplementedError


Row = TypeVar("Row", covariant=True)
RowFactory: TypeAlias = Callable[[Sequence[Any]], Row]

class Connection(Generic[Row]):
    @classmethod
    def connect(cls, *, row_factory: RowFactory[Row] = tuple_row) -> Self:
        raise NotImplementedError


reveal_type(Connection.connect())
reveal_type(Connection.connect(row_factory=tuple_row))
reveal_type(Connection.connect(row_factory=dict_row))

However, the typing system doesn't understand that if I don't specify a row_factory the type should be the type of the default. Running mypy 1.8 on this script:

testtype-default.py:21: error: Incompatible default for argument "row_factory" (default has type 
  "Callable[[Sequence[Any]], tuple[Any, ...]]", argument has type "Callable[[Sequence[Any]], Row]")  [assignment]
testtype-default.py:25: note: Revealed type is "testtype-default.Connection[Never]"
testtype-default.py:26: note: Revealed type is "testtype-default.Connection[builtins.tuple[Any, ...]]"
testtype-default.py:27: note: Revealed type is "testtype-default.Connection[builtins.dict[builtins.str, Any]]"

AFAICS the default for row_factory not only has a type compatible with the argument (as it is accepted as explicit value, in line 26), but it has enough information to convey the type, like the explicit values do. So I would expect line 25 to be revealed the same way as in line 26.

I think this would be useful because, at least in our use case, it would allow to implement the generic factory method with a default that we describe by using just Self, and would allow us to avoid the horrible re-definitions we need in subclassing. We actually wouldn't need to overload the method anymore.

On a hunch, it seems simpler than allowing a parametrized self as described in #1555.

[Daverball]

You are essentially asking for an automatic inference extension to PEP696 which has been delayed in favor of PEP695 (there was a push to have them both be accepted as part of 3.12, but the steering council ultimately decided against that)

I don't think that's a particularly good idea, I think it makes more sense to define the default explicitly on the TypeVar rather than rely on the type checker to infer it, especially considering that in type stubs (i.e. pyi files) non-simple default values are generally excluded. For now overloads are still the way to go to handle binding type vars for default arguments in generics, even though it does unfortunately come with the described limitations when returning Self.

To avoid introducing runtime overhead from the extra method calling super you can define the method inside an if TYPE_CHECKING: block, then it also doesn't need an implementation. That's how I usually deal with this case.

You could also return Connection[Row] instead of Self, which is also not ideal for inheritance, but for some subclasses of Connection you may not care and then you can skip redefining the method.

[dvarrazzo]

Thank you @Daverball: PEP696 seems indeed that it would cover the use case proposed.

[Viicos]

You are essentially asking for an automatic inference extension to PEP696 which has been delayed in favor of PEP695

@dvarrazzo in the meanwhile you can play with TypeVar defaults with pyright and see how it plays

[dvarrazzo]

As far as I can see, PEP 696 does exactly what intended, and it is already supported in mypy and current Python versions using typing_extensions >= 4.4

This modified version of the definition:

from typing import Any, Callable, Generic, Sequence, TypeAlias
from typing_extensions import Self, TypeVar

TupleRow: TypeAlias = tuple[Any, ...]

def tuple_row(values: Sequence[Any]) -> TupleRow:
    raise NotImplementedError


DictRow: TypeAlias = dict[str, Any]

def dict_row(values: Sequence[Any]) -> DictRow:
    raise NotImplementedError


Row = TypeVar("Row", covariant=True, default=TupleRow)
RowFactory: TypeAlias = Callable[[Sequence[Any]], Row]

class Connection(Generic[Row]):
    @classmethod
    def connect(cls, *, row_factory: RowFactory[Row] | None = None) -> Self:
        raise NotImplementedError


reveal_type(Connection.connect())
reveal_type(Connection.connect(row_factory=tuple_row))
reveal_type(Connection.connect(row_factory=dict_row))

is revealed as:

typing-5556.py:25: note: Revealed type is "typing-5556.Connection[builtins.tuple[Any, ...]]"
typing-5556.py:26: note: Revealed type is "typing-5556.Connection[builtins.tuple[Any, ...]]"
typing-5556.py:27: note: Revealed type is "typing-5556.Connection[builtins.dict[builtins.str, Any]]"