Extensible Typed Record possible in pyright, is it a hack?

[betafcc]

TL;DR:

• noticed a way to do 'type intersection operation' (TS &) in pyright, by being careful about bounding order
• doesn't look hacky to a fellow typescripter but might be to python
• not sure if issue worthy but don't know what is the proper channel to clarify

Tried reddit and discord, also don't think it's on mypy scope, maybe on typing if I'd be asking it as a feature but idk, pyright seems to be rolling a superset of it's own and I just wanna know if it's intended effect here

---

AFAIK there is no official way to make extensible typed record or TS-like intersection (& operator) in mypy/pyright, meaning, you can't define a function like:

T = TypeVar("T", bound=TypedDict)

def insert(d: T, key: K, value: V):
    return d | {key: value}

In a way that would keep the type info on the resulting dict.

Still there is an approach that I'm almost sure is legal: instead of keeping track of the dict, we keep track of key-value pair type, so we can do element type union to achieve collection type intersection (as in oop/subtying intersection):

from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Generic, TypeVar
from typing_extensions import LiteralString

M = TypeVar("M", bound=tuple[LiteralString, Any])
N = TypeVar("N", bound=tuple[LiteralString, Any])

@dataclass(frozen=True)
class Record(Generic[M]):
    dict: dict[Any, Any]

    def __or__(self, other: Record[N]) -> Record[M | N]:
        return Record(self.dict | other.dict)

    def __add__(self, other: N) -> Record[M | N]:
        return Record(self.dict | {other[0]: other[1]})

Now it works as expected:

from typing import NoReturn

empty = Record[NoReturn]({})

r = empty + ("a", 1) + ("b", "hi") + ("c", 3.14)
# (variable) r: Record[tuple[Literal['a'], Literal[1]] | tuple[Literal['b'], Literal['hi']] | tuple[Literal['c'], float]]

s = empty + ("d", True) + ("e", 2)
# (variable) s: Record[tuple[Literal['d'], Literal[True]] | tuple[Literal['e'], Literal[2]]]

rs = r | s
# (variable) rs: Record[tuple[Literal['a'], Literal[1]] | tuple[Literal['b'], Literal['hi']] | tuple[Literal['c'], float] | tuple[Literal['d'], Literal[True]] | tuple[Literal['e'], Literal[2]]]

Hopefully nothing hacky so far. The problem is retrieving a typed value from key, the closest solution I found was doing pattern matching:

def getval(self: Record[M], key: LiteralString) -> M:
    return key, self.dict[key]

match getval(rs, "c"):
    case ("c", cval): reveal_type(cval)  # Revealed type is 'float'
    case _: pass

Again, hopefully nothing hacky so far. But yeah, doesn't seem to be possible to do it in a generic K = TypeVar('K', bound=LiteralString) key variable.

But then I noticed, by carefully ordering typevar binding order, it actually is possible, although not in a instance bound method but in a curried function (or in a generic callable class):

K = TypeVar("K", bound=LiteralString)
V = TypeVar("V")
AnyPair = tuple[LiteralString, Any]

def get(key: K):
    def _get(record: Record[tuple[K, V] | AnyPair]) -> V:
        return record.dict[key]
    return _get

aval = get("a")(rs) # (variable) aval: Literal[1]
bval = get("b")(rs) # (variable) bval: Literal['hi']
cval = get("c")(rs) # (variable) cval: float
dval = get("d")(rs) # (variable) dval: Literal[True]
eval = get("e")(rs) # (variable) eval: Literal[2]

notpresent = get("notpresent")(rs) # (variable) notpresent: Unknown

And the line Record[tuple[K, V] | AnyPair] is what I wanna know if is hacky

So, is it? I'm matching against a union, in a way that does make sense to me, especially if I think about the M in class Record[M] as covariant, I mean, never went looking for it but I assume A | Any <: A | B | C | D | E for mypy/pyright/typescript, the only doubt is if the resolving of A | Any collapses to Any before that type-level pattern matching is done.

Also just to be sure, the manipulation of typevar bounding order, is it ok to count on it in general? Or is it just a unintended side effect of pyright's type inference? That feature I've been using quite a lot in other more 'normal' code, so I was counting on it being ok.

More complete and careful implementation at the gist

[betafcc]

Also there is another way I found that does not use that hack, but I'm waiting a clarification from mypy to know if user-defined type operators are intended as feature.

It has two assumptions:

1. F[A | B | C | D] <-> F[A] | F[B] | F[C] | F[D]
2. A | NoReturn <-> NoReturn | A <-> A

With those we can:

from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Generic, Literal, NoReturn, Type, TypeVar, overload
from typing_extensions import LiteralString

K = TypeVar("K", bound=LiteralString)
V = TypeVar("V")

@dataclass(frozen=True)
class Getter(Generic[K]):
    key: K

    @overload
    def __getitem__(self, record: Type[tuple[K, V]]) -> Type[V]:  ...
    @overload
    def __getitem__(self, record: Type[tuple[Any, Any]]) -> NoReturn: ...
    def __getitem__(self, record: Any) -> Any:  ...

Got = Getter("foo")[
    tuple[Literal["hi"], int] | tuple[Literal["bye"], str] | tuple[Literal["foo"], bool]
]
# (type alias) Got: Type[bool]

By collapsing all non-matching values to NoReturn, we get an union of only the elements that match the key, while keeping the type info of the values

[erictraut]

Converting this to a discussion topic since it's a question, not a bug report.

[erictraut]

A few thoughts...

Pyright is a standards-based type checker. It doesn't support non-standard type system features. Any new additions to the Python type system would need to go through the standards process, which typically involves discussion on the typing discussion forum or typing-sig email list followed by a draft PEP.

While pyright conforms to the Python typing standard, there are many aspects of the type system that are not well defined or are intentionally undefined (left up to each type checker). Examples include type inference behaviors and rules for choosing from among multiple valid TypeVar solutions.

Your solution above relies on assumptions that are tenuous at best. The Python type system spec (PEP 484, etc.) provide little or no guidance about how type variables are supposed to be solved. There are often many solutions that are all "correct" from a type standpoint, but one solution is often preferable to others. This comes up frequently when matching against unions that involve type variables, as seen in your first sample above. Pyright contains a bunch of internal heuristics that I've tuned over multiple years that attempt to produce the most desirable solution in the majority of cases. As with any heuristics, they are not infallible, and the behavior may change as the heuristics evolve. If your code is relying on such heuristic-driven behavior for key type checking capabilities, then you're on thin ice. I wouldn't recommend using the above tricks in any public code bases or production code that needs to work with multiple type checkers or is intended to be maintained over time.

I've updated your function get above so it has a declared return type. While pyright is able to infer return types, mypy cannot, and I was interested to see what mypy would do with this code.

def get(key: K) -> Callable[[Record[tuple[K, V] | AnyPair]], V]:
    ...

As I predicted, mypy evaluates the type of get("a")(rs) differently than pyright. Mypy leaves V unsolved because it matches the argument rs against the AnyPair portion of the union rather than the tuple[K, V] portion. Mypy's behavior is defensible here, as is pyright's.

aval = get("a")(rs) # mypy: "error: need type annotation for avail"
reveal_type(aval) # pyright: Literal['hi'], mypy: Any

TypeScript doesn't run into these ambiguities because there is a single reference TypeScript compiler implementation. If there were multiple competing TypeScript implementations, the TypeScript type spec would need to be much more complete and pin down a bunch of behaviors that are implicit today.

In your final example above, you're attempting to define a type alias Got. Traditional (PEP 484-style) type aliases in Python are squirrelly because they are difficult for a type checker to distinguish from a variable assignment expression. Most (all?) other languages that support static typing provide a dedicated keyword for defining type aliases (type in TypeScript, for example). PEP 613 tried to address this shortcoming (in a rather unfortunate manner, IMO) by adding a special form TypeAlias. If you add a TypeAlias annotation to your Get expression, you'll find that pyright now flags it as an illegal type alias because you're using an expression form (a call expression) on the RHS that is not legal for type expressions. (You may be interested to learn that I'm proposing to add a type keyword to Python 3.12 — see PEP 695). My main point here is that the definition of Got should be treated as illegal because it's not a proper type alias. Pyright lets you get away with this because it doesn't strictly enforce the type alias rules for traditional type aliases, but you shouldn't rely on this behavior.

The concept of a user-defined type operator does not appear in the Python type system today. If you think there is a compelling need for this and you are so inclined, you could champion a proposal for an addition to the type system.

[betafcc]

Thanks for your attention on a sunday evening, also thanks for the amazing tool.

---

Converting this to a discussion topic since it's a question, not a bug report.

thanks, a bit newbie here

---

Pyright is a standards-based type checker.

Sure, didn't mean to counter, just noticed pyright has finer inference in many cases, allowing more powerful compositions. Eg, in mypy even the non-hacky Record methods need some explicit cast to work

---

Any new additions to the Python type system

Not really considering new features, more wanting to know if some uses I found are ok to use. In fact, I'd even prefer not to have some of the crazier TS features here as code get's unreadable in many libs and I personally get a bit obssessive and improductive trying to put as many operations in lint-time as possible.

Still, Python is annoyingly and uncharacteristically verbose on everything related to type hints, so anything I can find to mitigate that, I wanna know if is legal to use

---

PEP 613 tried to address this shortcoming (in a rather unfortunate manner, IMO)

and

You may be interested to learn that I'm proposing to add a type keyword to Python 3.12

Yeah, I've been using your 3.12 draft more than I should, was suggested by pylance to allow typeand def foo[A, B], really feel like cpython is delaying the unavoidable standardization of proper type-level-only syntax

---

If you add a TypeAlias annotation to your Get expression, you'll find that pyright now flags it as an illegal type alias

Noticed. Was trying to find a way to order the typevar binding in a way that is allowed, I need to bind K before the rest, so the result resolves as desired, still, don't seem that conceptually it needs to mix type-level code with runtime-level code, just couldn't find how

---

if you think there is a compelling need for user defined operators

Well for one this kind of stuff on python/typing issue you were just now, becomes possible without mypy extensions (notice can be thought as a use for this record without K bound to string plus a KeyOf operator that is possible even in mypy), in fact, together with pyrights finer tuned features, I'm pretty sure HKT is already possible by using the well known Lightweight higher-kinded polymorphism encoding (replicated via extensions by https://github.com/dry-python/returns and on TS by https://github.com/gcanti/fp-ts)

But as I mentioned, my intent is more to be sure if what is working is legal, with the only goal of having a slightly more terse and composable syntax, also to get more knowledge in this domain as I don't feel in position of proposing anything solid.

---

In any case I feel my doubt is answered, and I can wait for a proper extensible typeddict for now, as my actual use case ended up being possible without those hacks, giving I was just mapping strings to strings, here in a gist if has any relevance.

Thanks again for your time and I'll be sure to use the proper channels next time