How can I type annotate a factory helper's **kwargs to match that of the __init__ of the type being created?

[cjw296]

So, for this code:

class Collection:
    def make(self, type_: Type[T], **attrs) -> T:
        ...

How can I annotate **attrs such that a type checker, mypy in particular, will correctly ensure that any keywords passed are a valid subset of those that could be passed to T's constructor?

[erictraut]

You can use a ParamSpec. There's no way to constrain it to only keyword parameters though. After all, the constructor may support positional parameters.

from typing import Callable

class Collection:
    def make[**P, T](
        self, type_: Callable[P, T], *args: P.args, **kwargs: P.kwargs
    ) -> T: ...

Here's the same thing using older (pre-3.12) syntax:

from typing import Callable, ParamSpec, TypeVar

P = ParamSpec("P")
T = TypeVar("T")

class Collection:
    def make(self, type_: Callable[P, T], *args: P.args, **kwargs: P.kwargs) -> T: ...

[cjw296]

You can have only *args using TypeVarTuple instead of ParamSpec, but there's currently no equivalent for mapping based variadic generics, so you can't have only **kwargs.

Well that sucks :-( I would have thought "only accepting kwargs" is a more common pattern over Python's history, so it's disappointing that's the option that's not supported :-/

Also type[T] is a subtype of Callable[P, T] where P represents the arguments of the constructor of T. I don't see a reason to arbitrarily restrict this function to type[T], unless you want to be able to use it in isinstance calls or do something else where you need an instance of type.

Yes, exactly that, this package needs a type, not some arbitrary callable. Once again python typing ends up burning cpu at runtime.

Pragmatically, I either leave it as:

from typing import Type, Any, TypeVar

T = TypeVar("T")

class Collection:
    def make(self, type_: Type[T], **kwargs: Any) -> T: ...

...or I have to:

from typing import Callable, ParamSpec, TypeVar

P = ParamSpec("P")
T = TypeVar("T")

class Collection:
    def make(self, type_: Callable[P, T], *args, P.args, **kwargs: P.kwargs) -> T:
         assert not args
         assert isinstance(type_, type)
         ...

sigh

[bzoracler]

Here's a contortion which works (practically) on mypy (Playground example):

from collections.abc import Callable
from typing_extensions import Any, Concatenate, ParamSpec, Protocol, TypeAlias, TypeVar


Self: TypeAlias = Any
T = TypeVar("T")
P = ParamSpec("P")


class KWArgsOnly(Protocol):
    def __call__(self, factory: type[Any], /, **kwargs: object) -> Any: ...


def reject_non_type_and_non_kwargs(
    make: Callable[Concatenate[Self, Callable[P, T], P], T],
) -> Callable[Concatenate[Callable[P, T], P], T] | KWArgsOnly:
    return make  # type: ignore[return-value]


class Collection:
    @reject_non_type_and_non_kwargs
    def make(self, factory: Callable[P, T], /, *args: P.args, **kwargs: P.kwargs) -> T:
        return factory(*args, **kwargs)


class Instance:
    def __init__(self, a: int, b: str, c: list[int]) -> None: ...


def non_type_factory(a: int, b: str, c: list[int]) -> Instance:
    return Instance(a, b, c)

reveal_type(Collection().make(Instance, a=1, b="", c=[]))  # OK
Collection().make(Instance, 1, b="", c=[])  # Error: passed positional argument
Collection().make(non_type_factory, a=1, b="", c=[])  # Error: passed non-type

The idea is that @reject_non_type_and_non_kwargs returns the make method's original signature unioned with a callback that only accepts a type and keyword arguments. This means that you'll get errors at the call site if you try to pass non_type_factory (a function) and/or positional arguments afterwards.

This doesn't protect you from making a bad signature for make though, so ensure that you have properly demarcated your positional prefix arguments from the constructor.

[Daverball]

That's a neat trick, however it changes the return type to Instance | Any, here's a slightly adjusted version that properly preserves the return type: https://mypy-play.net/?mypy=latest&python=3.12&gist=63473b4b5ef209952fa5ea5c1f2ba7d5

[cjw296]

It really does sound like there's a deficiency in Python's typing, where ParamSpec should be usable to say, either args or kwargs or a way of saying ParamSpec(...partial=True) or some such. Has that discussion been has? If so, where?

If not, where should it be had?

The fact that partial is still typed as Any in typeshed seems bad to me :-(

[Daverball]

The Python Discourse would be the current best place for typing related discussions: https://discuss.python.org/

ParamSpec adjacent topics and discussions have come up several times. Python function signatures are very versatile, but that also makes them very difficult to deal with in static typing contexts. One big issue is expressiveness, since we currently have no way to spell a complex signature in a pure type expression. We instead have to rely on callback protocols, which only gives us part of the functionality we need and some things are only very difficult or awkward to express. This is a relative recent discussion about a possible syntax extension to fix that part of the problem: https://discuss.python.org/t/reviving-the-hybrid-keyword-arrow-syntax-for-callables/52499

Once we can actually express all kinds of signatures in type expressions, it will be much easier to add the bound argument to ParamSpec, which would allow your use-case (although you'd still have to write *args: P.args, **kwargs: P.kwargs). TypeVarMapping has been discussed several times as well, but after how difficult ParamSpec and TypeVarTuple turned out to implement and how long it took for even just basic support to land in more than one type checker, the brakes were hit on adding more kinds of type parameters.

The main focus for Python typing currently is to write a complete typing spec and compliance test suite, so type checkers have to waste less time individually figuring out how certain features are supposed to work, because the wording in the original PEP was too loose/ambiguous.