Maybe bug: Creating diamond diagram using NewType

[Jacob-Stevens-Haas]

I tried to use NewType to create a diamond-inheritance subtype, similar to the following

Foo = NewType("Foo", int | str)

As mentioned in #1397, that's verboten. And indeed, pyright/pylance say Expected class as second argument to NewType. But why is that the case? The second argument to NewType doesn't receive much detail in the docs (also), other than referring to it as a type. And PEP 483 says

Union[str, int] is a type but not a proper class

(although perhaps typing docs mangles the distinction between classes and types with "treat the new type as if it were a subclass [not subtype?] of the original type")

Am I using the wrong tool for the job? Is there not currently a way to create a diamond-diagram subtype for type checking without creating an actual subclass?

[erictraut]

When you create a NewType, the resulting type is supposed to act as though it's a class that derives from the specified type. That means the second argument must be a class. It can't be a union — just like you can't use a union as a base class in a class statement.

class Foo(int | str): ... # Illegal!

Keep in mind that unions are not ordered, so int | str is the same type as str | int. By comparison, when you use multiple inheritance, the order of the base classes is significant because it affects the mro.

# These are very different classes!
class Sub1(int, str): ...
class Sub2(str, int): ...

I don't understand what you're trying to achieve by creating "a diamond-diamond subtype for type checking without creating an actual subclass". There's potentially another way to accomplish your goal, but you'd have to explain more for me to offer a suggestion.

[Jacob-Stevens-Haas]

Ah yes, I'm familiar that users would need to explicitly construct - but the user in this case is me in a few months after i'm working on a different part of the code, or close colleagues working on the same internal-use package, or inside private modules in a public package. And thanks - lightweight wrappers could be useful in cases.

I wanted to return and ask about something you said:

When you create a NewType, the resulting type is supposed to act as though it's a class that derives from the specified type.

I'm confused by what it means for a type to "act" as anything. If I understand correctly (and I'm sure I don't), types in type checking only "act" in the sense that they obey a subtype relation. At runtime, the object acts as it is, since it is unchanged. Which is why I'm confused by why MRO would matter.

[erictraut]

By "act as", I mean that you can use all of the same methods and attributes that are available for the class.

MRO would matter if NewType were allowed to be created with more than one class, as you were attempting to do. When multiple classes are involved — especially when those classes both implement methods or attributes with the same name, MRO determines which class overrides and which is overridden.

[Jacob-Stevens-Haas]

Thanks for the follow up, Eric! I understand why MRO is important for classes, but not for types. In Derived = NewType("Derived", tp), the Derived object is of class NewType, with no relation to the MRO of tp. Derived.__call__ is the identity function. tp is assigned toDerived.__supertype__. NewType defines an __mro_entries__ method, but only for simplifying errors if someone accidentally subclasses NewType. So I'm wondering if the prohibition of NewType(name, Union[int, str]) is merely a consequence of wanting to prevent class Foo(NewType). (source)

I guess I'm asking, is the restriction of tp to being a class in NewType(name, tp) done for implementation reasons, even though the type theory concept is valid?

Or am I misunderstanding the association of python typing syntax to typing theory in PEP 483? I had believed the role of NewType was to declare an arbitrary subset (subtype) of a larger set (type). A Union of sets (types) is also set (type). It's legal to take an arbitrary subset (subtype) of any larger set (type). Am I wrong there?

[erictraut]

I had believed the role of NewType was to declare an arbitrary subset (subtype) of a larger set (type).

No, that's not the role of NewType. It's meant to be a stand-in (a proxy) for a specified class. At runtime, the NewType is erased completely, and it's the same as the class it is proxying. At static analysis time, the NewType is treated as a subclass of the specified class. This allows you to statically distinguish between specific uses of the class being proxied. For example, let's say that your function accepts two float values -- the first represents a "height" value and the second a "width" value. If you want to help the caller avoid confusing the two, you can do the following:

Height = NewType("Height", float)
Width = NewType("Width", float)

def computeArea(h: Height, w: Width) -> float:
    return h * w

computeArea(1.0, 1.4) # Error
computeArea(Height(1.0), Width(1.4)) # Works

[Jacob-Stevens-Haas]

Oh, ok, that makes sense. Thanks for clarifying, and thanks for sticking with me through this question. :)

[Jacob-Stevens-Haas]

Relatedly, I've also found a bit about typing.Protocol. That comes close, because scipy.sparse.spmatrix and numpy.ndarray are the only classes I know that implement python's @ operator via __matmul__. However, Pyright also doesn't like using NewType around a Protocol. If the Intersection types considered in PEP 483 were implemented in Python, I could achieve what I'm trying to with the intersection of a protocol for __matmul__ with a NewType(Foo, object).

[erictraut]

Pyright also doesn't like using NewType around a Protocol

Protocols cannot be instantiated, so it wouldn't make sense to use them with NewType. There would be no way to instantiate the NewType because the constructor for a NewType requires you to pass an instance of the class. Not surprisingly, mypy also emits an error if you attempt to use NewType with a protocol, a union, or anything other than a normal (nominal) class.